-
Notifications
You must be signed in to change notification settings - Fork 237
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chunked: various improvements #1092
Conversation
17460df
to
2d47880
Compare
@cgwalters PTAL It is still a draft as it is barely tested, but I think this could help with the deduplication of the large Go binaries we were discussing. The deduplication, in this case, would be just for the network data since locally it is not usable as reflinks require the data to be aligned. |
989f9c0
to
98c459b
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you toss up some sort of even brief description or diagram (e.g. https://asciiflow.com/#/ ) of how the chunks are stored? How does it relate to the container layers, etc? It's hard to just jump in and look at code without that.
Also, it'd really help build confidence in this to have even some basic unit tests. See e.g. https://github.com/ostreedev/ostree/blob/main/tests/test-rollsum.c
The ostree static delta code which uses this is also tested by unit tests that verify that it regenerates exactly the same files.
98c459b
to
d5ffb23
Compare
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
d5ffb23
to
432e3b9
Compare
yes, I need to write some documentation about it. The metadata JSON file is based on the same format as estargz: https://github.com/containerd/stargz-snapshotter/blob/main/docs/estargz.md, chunks are represented using the estargz TOC file. For the tarball itself, there is not much difference, the only difference is that the compression must be restarted when a chunk is created, so we can pick the offset and use it for generating the TOC JSON metadata file that is appended at the end of the tarball. Locally there is no difference in how the files are stored. We lookup the layers metadata to know what files/chunks are present. If a chunk is already present locally, we read directly from the checked-out file. e.g. we have the following entries in the metadata for a layer present in the local store:
if we pull a new image that has a chunk with the digest Loading the JSON metadata for each layer in the storage is quite expensive, this is something that could be improved, but I've not looked into it yet. I've done some tests with appending data to a big file, and the rolling checksum seems to help a lot. Unfortunately, I don't see a big improvement with Go binaries (I've tried with |
b92cb8e
to
5dd63b3
Compare
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
it solves a problem where the discard could be performed before the compression handler was closed (through a deferred call). Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
0795a4d
to
45e6852
Compare
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
b4340ae
to
c2defb9
Compare
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
avoid using slices. I've seen a drop of ~20M in memory usage with a fedora image. Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
a reference to the just created parent directory is already opened, so use it directly. Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
avoid parsing each json TOC file for the layers in the local storage, but attempt to create a lookaside cache in a custom format faster to load (and potentially be mmap'able). The same cache is used to lookup files, chunks and candidates for deduplication with hard links. There are 3 kind of digests stored: - digest(file.payload)) - digest(digest(file.payload) + file.UID + file.GID + file.mode + file.xattrs) - digest(i) for each i in chunks(file payload) Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
reduce the number of allocations done by the parser by reading into a bytes.Buffer. Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
c2defb9
to
812c8bd
Compare
LGTM |
various improvements for the pkg/chunked package.
Noteworthy:
More details in each commit.
Signed-off-by: Giuseppe Scrivano gscrivan@redhat.com