Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for exporting nydus compression type #2581

Merged
merged 6 commits into from
Nov 8, 2022

Commits on Nov 1, 2022

  1. cache: support nydus compression type

    Nydus image is a container accelerated image format provided by the
    Dragonfly image-service project, which offers the ability to pull
    image data on demand, without waiting for the entire image pull to
    complete and then start the container. It has been put in production
    usage and shown vast improvements over the old OCI image format in
    terms of container launching speed, image space, and network bandwidth
    efficiency, as well as data integrity. Nydus image can be flexibly
    configured as a FUSE-based user-space filesystem or in-kernel
    EROFS (from Linux kernel v5.16) with Nydus daemon in user-space,
    integrating with VM-based container runtime like KataContainers
    is much easier.
    
    Nydus has provided a conversion tool Nydusify for converting OCIv1
    image to Nydus image and integrated into Harbor Acceld as a conversion
    driver, which assumes that the OCI image is already available in the
    registry, but a better way would be to build the Nydus images directly
    from the build system instead of using the conversion tool, which would
    increase the speed of the image export, so we experimentally integrated
    the Nydus export in Buildkit.
    
    Unlike other compression formats (gzip, estargz, etc.) in OCI image,
    nydus is divided into two types of layer, blob, and bootstrap, where
    blob serves as the data part of each layer of the image, and bootstrap
    serves as the metadata of the whole image, the bootstrap is equivalent
    to the view of the whole image filesystem after all layers overlay. For
    example, for an OCI image with 3 layers, the corresponding nydus image
    is 4 layers (3 layers of blob + 1 layer of bootstrap).
    
    The nydus-snapshotter project provides a package to do the actual layer
    compression, this commit imports the package to implement the export of
    nydus compression type.
    
    Signed-off-by: Yan Song <imeoer@linux.alibaba.com>
    imeoer committed Nov 1, 2022
    Configuration menu
    Copy the full SHA
    cf70ab6 View commit details
    Browse the repository at this point in the history
  2. exporter: append extra layer for nydus compression type

    For nydus image, appending an extra nydus bootstrap layer
    to the manifest, this layer represents the whole metadata
    of filesystem view for the entire image.
    
    Nydus uses this extra boostrap layer to reduce the number
    of HTTP request and FUSE overlay's overhead on preparing
    the container's rootfs.
    
    For now we just support this in image type exporter.
    
    Signed-off-by: Yan Song <imeoer@linux.alibaba.com>
    imeoer committed Nov 1, 2022
    Configuration menu
    Copy the full SHA
    0d1f8c4 View commit details
    Browse the repository at this point in the history
  3. test: add test for nydus compression type

    Signed-off-by: Yan Song <imeoer@linux.alibaba.com>
    imeoer committed Nov 1, 2022
    Configuration menu
    Copy the full SHA
    fa43b9f View commit details
    Browse the repository at this point in the history
  4. vendor: sync for supporting nydus compression type

    Signed-off-by: Yan Song <imeoer@linux.alibaba.com>
    imeoer committed Nov 1, 2022
    Configuration menu
    Copy the full SHA
    f53eb59 View commit details
    Browse the repository at this point in the history
  5. doc: add usage for nydus compression type

    Introduce nydus image and show that how to export nydus image, and
    add known limitations.
    
    Signed-off-by: Yan Song <imeoer@linux.alibaba.com>
    imeoer committed Nov 1, 2022
    Configuration menu
    Copy the full SHA
    ab6e8bb View commit details
    Browse the repository at this point in the history

Commits on Nov 3, 2022

  1. nydus: adjust code structure for independence

    Signed-off-by: Yan Song <imeoer@linux.alibaba.com>
    imeoer committed Nov 3, 2022
    Configuration menu
    Copy the full SHA
    3e9e898 View commit details
    Browse the repository at this point in the history