container_commit: Add support for converting writable layers to eStargz blobs #4351
Description
What is the problem you're trying to solve
Current Limitation
The current nerdctl commit implementation does not support converting the container's writable layer into eStargz format blobs. This means that the commited layer in the commited image cannot benefit from lazy pulling optimizations, even when other image layers are already in eStargz format.
Problem Statement
When committing a container to create a new image:
The writable layer (containing all changes made to the container) is converted to a regular tar.gz blob
This prevents the resulting image from supporting lazy pulling
Large applications or data added to the container cannot benefit from on-demand loading
Container startup performance degrades, especially for images with significant changes in the writable layer
# skopeo inspect --raw docker://registry.domain.local/library/app:estargz-after-commit
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"schemaVersion": 2,
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"digest": "sha256:6ebaca2e3e88f3a52f8aeebf02b1757b8fefe6ec646da95b67fe639d092deb91",
"size": 7763
},
"layers": [
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:9a8ece1256563321005c05a5e56e74f9b6b5033fd3b278156d5af768f2343b72",
"size": 31413787,
"annotations": {
"containerd.io/snapshot/stargz/toc.digest": "sha256:e127ce321dc5c5bc6d719142ceba4f0fc3e5c802014bd849eaea6c04d3615cf8",
"io.containers.estargz.uncompressed-size": "79133696"
}
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:d2282c21a5b83b270fb403ff6ea557ff4c70ab0088e08bad9ae55018610cd2c4",
"size": 3570041,
"annotations": {
"containerd.io/snapshot/stargz/toc.digest": "sha256:dec97136099e10d4df7c964cfb34f372119ac1f3b03a89b2ba76a7f8f60c2ab9",
"io.containers.estargz.uncompressed-size": "9690624"
}
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:0e3f6307718ed3ef3d1cb7bfd6edf4f3dcd08d1ee76d0133e6117c7046b450b1",
"size": 12415119,
"annotations": {
"containerd.io/snapshot/stargz/toc.digest": "sha256:2b7da3c3c47c4307da650d0e1c6cf26403eaee73d88fc9af2506fc2904fa3a16",
"io.containers.estargz.uncompressed-size": "32730112"
}
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:57cf968ef8c64430016a68df159fd96177e0e41ec7cfc16be3353929743acf31",
"size": 1258,
"annotations": {
"containerd.io/snapshot/stargz/toc.digest": "sha256:94c00d8e6297b18f26351a002574763f37873cb44d6668361306ee6a93477b78",
"io.containers.estargz.uncompressed-size": "7680"
}
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:8496fa43574a0f7617c3af671bad8d7b85dd8ef9419d35b795de47d1c149a84a",
"size": 3059422,
"annotations": {
"containerd.io/snapshot/stargz/toc.digest": "sha256:8047acf04ec542eaa2a80de310e07bba2f7a628014313bc21301e77bd872954b",
"io.containers.estargz.uncompressed-size": "10897408"
}
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:d17aeaf28cec17050a8ddd551495e83e4fbd6027232855bd1b244f9e42cb2cf0",
"size": 1837077,
"annotations": {
"containerd.io/snapshot/stargz/toc.digest": "sha256:8c3b17497114e5dfc0336cd1807c0e90359616fa2f76ea8e4814778ee857c909",
"io.containers.estargz.uncompressed-size": "4493824"
}
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:71d29537527e51c8b7c3037a9aacdaab5fd09ec7a5b5f88efb75fe03900eda2e",
"size": 808812211,
"annotations": {
"containerd.io/snapshot/stargz/toc.digest": "sha256:a2d7960cd75fdd2a41d758170bdf67307ab04cf8ae3d02e89c7758ca464b4b53",
"io.containers.estargz.uncompressed-size": "1073745920"
}
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:16b3c6368de8e6942263a0e9e34d7aaffc79a7f761b65b3f33020e3748a7ca55",
"size": 156
}
]
}
Describe the solution you'd like
Extend the nerdctl commit command to support eStargz conversion for writable layers:
# Enable eStargz conversion for the writable layer during commit
nerdctl commit --format oci --estargz e8c42a39aca4 registry.domain.local/library/app:estargz-after-commit
# With additional eStargz configuration options
nerdctl commit --format oci --estargz \
--estargz-compression-level=9 \
--estargz-chunk-size=16MB \
--estargz-min-chunk-size=4MB \
e8c42a39aca4 registry.domain.local/library/app:estargz-after-commit
Additional context
No response
Metadata
Metadata
Assignees
Type
Projects
Milestone
Relationships
Participants
Activity
commit: support estargz conversion with writable layer in container c…
commit: support estargz conversion with writable layer in container c…
commit: support estargz conversion with writable layer in container c…
commit: support estargz conversion with writable layer in container c…
commit: support estargz conversion with writable layer in container c…
commit: support estargz conversion with writable layer in container c…
commit: support estargz conversion with writable layer in container c…
commit: support estargz conversion with writable layer in container c…
commit: support estargz conversion with writable layer in container c…