-
Notifications
You must be signed in to change notification settings - Fork 234
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support of sparse files in container images #1091
Comments
It sounds like a useful feature, and we can probably extend our format to support them (I've not looked though into it, so I am not sure how difficult it would be with the existing Go libraries for handling tarballs). Should the holes to be automatically detected (like |
It seems an old missing feature in go: golang/go#13548 |
could we circumvent the limitation and inject the missing metadata in the tar header? |
I haven't dug too much into the technical alternatives. However, I think the main issue might be that we need to have the ability to write the missing metadata at build time and read it when we pull and extract the layer. So, this depends on which tool we used to build the image. |
Hi, |
I think we could add it as part of the zstd:chunked format based on top of: #1092 I don't think it is worth trying to add this information to the tar stream itself, as anyway the compression deals with sparse files quite well. |
automatically detect holes in sparse files (the threshold is hardcoded at 1kb for now) and add this information to the manifest file. The receiver will create a hole (using unix.Seek and unix.Ftruncate) instead of writing the actual zeros. Closes: containers#1091 Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
I've opened a PR for sparse files support in zstd:chunked: #1102 I think this is the best we can achieve with the current container images. The holes are automatically detected. The threshold is currently hardcoded to 1kb. So it doesn't "preserve" the existing information, but this would require a lot of changes in the container tools, because we first create the tar stream and pass it around without any access to the original file additional metadata (including holes). @alicefr is it fine for your use case? A simple test:
the generated layer:
and from the receiver side:
|
@giuseppe I'll give a try and let you know :) thanks |
automatically detect holes in sparse files (the threshold is hardcoded at 1kb for now) and add this information to the manifest file. The receiver will create a hole (using unix.Seek and unix.Ftruncate) instead of writing the actual zeros. Closes: containers#1091 Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
automatically detect holes in sparse files (the threshold is hardcoded at 1kb for now) and add this information to the manifest file. The receiver will create a hole (using unix.Seek and unix.Ftruncate) instead of writing the actual zeros. Closes: containers#1091 Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
automatically detect holes in sparse files (the threshold is hardcoded at 1kb for now) and add this information to the manifest file. The receiver will create a hole (using unix.Seek and unix.Ftruncate) instead of writing the actual zeros. Closes: containers#1091 Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
automatically detect holes in sparse files (the threshold is hardcoded at 1kb for now) and add this information to the manifest file. The receiver will create a hole (using unix.Seek and unix.Ftruncate) instead of writing the actual zeros. Closes: containers#1091 Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
automatically detect holes in sparse files (the threshold is hardcoded at 1kb for now) and add this information to the manifest file. The receiver will create a hole (using unix.Seek and unix.Ftruncate) instead of writing the actual zeros. Closes: containers#1091 Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
automatically detect holes in sparse files (the threshold is hardcoded at 1kb for now) and add this information to the manifest file. The receiver will create a hole (using unix.Seek and unix.Ftruncate) instead of writing the actual zeros. Closes: containers#1091 Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
automatically detect holes in sparse files (the threshold is hardcoded at 1kb for now) and add this information to the manifest file. The receiver will create a hole (using unix.Seek and unix.Ftruncate) instead of writing the actual zeros. Closes: containers#1091 Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Today, it isn't possible to preserve the sparse files if they are copied inside a container image. This is reported as a limitation in the OCI spec. A similar issue has been reported in buildkit. Do we have any chances in the future to support this?
My particular use case affects KubeVirt and container disks. I reported the issue in kubevirt/kubevirt#6976 . In KubeVirt , we use container images to ship VM images and the spare files allow us to reduce the final container image size. I understand that this is a very specific use case but the feature might be useful for other situations.
The text was updated successfully, but these errors were encountered: