chunked: various improvements

[giuseppe]

various improvements for the pkg/chunked package.

Noteworthy:

• support splitting a file into multiple chunks and perform network deduplication.
• local metadata cache for each layer. It avoids parsing the json metadata file for each layer in the storage.
• use valyala/gozstd instead of github.com/klauspost/compress. It is much faster.
• copy local files from multiple goroutines.
• use json-iterator to improve JSON manifest parsing
• reuse a static buffer for io operations

More details in each commit.

Signed-off-by: Giuseppe Scrivano gscrivan@redhat.com

[giuseppe]

@cgwalters PTAL

It is still a draft as it is barely tested, but I think this could help with the deduplication of the large Go binaries we were discussing.

The deduplication, in this case, would be just for the network data since locally it is not usable as reflinks require the data to be aligned.

[cgwalters]

Can you toss up some sort of even brief description or diagram (e.g. https://asciiflow.com/#/ ) of how the chunks are stored? How does it relate to the container layers, etc? It's hard to just jump in and look at code without that.

Also, it'd really help build confidence in this to have even some basic unit tests. See e.g. https://github.com/ostreedev/ostree/blob/main/tests/test-rollsum.c

The ostree static delta code which uses this is also tested by unit tests that verify that it regenerates exactly the same files.

[giuseppe]

Can you toss up some sort of even brief description or diagram (e.g. https://asciiflow.com/#/ ) of how the chunks are stored? How does it relate to the container layers, etc? It's hard to just jump in and look at code without that.

Also, it'd really help build confidence in this to have even some basic unit tests. See e.g. https://github.com/ostreedev/ostree/blob/main/tests/test-rollsum.c

The ostree static delta code which uses this is also tested by unit tests that verify that it regenerates exactly the same files.

yes, I need to write some documentation about it.

The metadata JSON file is based on the same format as estargz: https://github.com/containerd/stargz-snapshotter/blob/main/docs/estargz.md, chunks are represented using the estargz TOC file.

For the tarball itself, there is not much difference, the only difference is that the compression must be restarted when a chunk is created, so we can pick the offset and use it for generating the TOC JSON metadata file that is appended at the end of the tarball.

Locally there is no difference in how the files are stored. We lookup the layers metadata to know what files/chunks are present. If a chunk is already present locally, we read directly from the checked-out file.

e.g. we have the following entries in the metadata for a layer present in the local store:

 {
      "type": "reg",
      "name": "usr/bin/podman",
      "mode": 493,
      "size": 49984904,
      "modtime": "2021-12-23T13:10:02+01:00",
      "accesstime": "0001-01-01T00:00:00Z",
      "changetime": "0001-01-01T00:00:00Z",
      "digest": "sha256:8270b870b26cdca16bc158f6d230a1d6ca1d3d7503b2f97d9bcf85dbb6327f1b",
      "offset": 131,
      "endOffset": 17374600,
      "chunkSize": 1625,
      "chunkDigest": "sha256:1b1a58f4ba739b5d076472a4559c15a7c31b20182c7e1abd5bb55761bd185a8b"
    },
    {
      "type": "chunk",
      "name": "usr/bin/podman",
      "offset": 1055,
      "chunkSize": 9188,
      "chunkOffset": 1625,
      "chunkDigest": "sha256:5d7700542a3069fecd4cdfc5f1e1f5f7f50f5109e78ff137267be199af52f724"
    },
    {
      "type": "chunk",
      "name": "usr/bin/podman",
      "offset": 4720,
      "chunkSize": 2799,
      "chunkOffset": 10813,
      "chunkDigest": "sha256:ebb089e3b60dac8a238d1154e9259efee9ffd298d3da1ba1d13a34c3eda9340a"
    },

if we pull a new image that has a chunk with the digest "chunkDigest": "sha256:ebb089e3b60dac8a238d1154e9259efee9ffd298d3da1ba1d13a34c3eda9340a" then we copy it from the file $LAYER_CHECKOUT/usr/bin/podman at offset 2799 for 2799 bytes.

Loading the JSON metadata for each layer in the storage is quite expensive, this is something that could be improved, but I've not looked into it yet.

I've done some tests with appending data to a big file, and the rolling checksum seems to help a lot. Unfortunately, I don't see a big improvement with Go binaries (I've tried with podman version 3.4.0 and podman version 3.4.4 and I get ~12% dedup if I use 13 bits for the rolling checksum, and ~%5 if using 16 bits as it is set now). With 13 bits, there are too many chunks generated and that has an impact on the final tarball size. I wonder if there is anything we can do with the linker settings to generate more similar binaries

[rhatdan]

LGTM