Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Containerd image and snapshot backend integration #38738

Draft
wants to merge 61 commits into
base: master
from

Conversation

@dmcgowan
Copy link
Member

commented Feb 14, 2019

Opening this up as a draft as there are many things left to do before this is mergeable.

Highlights

  • Image and reference store deprecated for containerd backend
  • ImageID can reference manifests, manifest lists, or image configs (allowing better support for multi-arch images)
  • Support for multiple unpacked storage backends (images point to their backend)
  • Multi-arch Image support

Major things left to do

  • Migration of existing image store data
  • Build code
  • Artifact creation on push and commit
  • Snapshot support as an option for images to point to
  • Many, many TODOs

Implements #38043

Does not include, both snapshotters and graphdriver interfaces will be supported in this PR

  • Graph drivers scoped down (no more mounting/diffing)
  • Graph drivers supported as containerd snapshotter plugins

Closes #26146
Closes #38043
Closes #33355
Closes docker/cli#122

@dmcgowan dmcgowan force-pushed the dmcgowan:containerd-integration branch from 537f342 to 481414a Feb 14, 2019

@AkihiroSuda

This comment has been minimized.

Copy link
Member

commented Feb 15, 2019

Is this planned to be merged before Docker v19.03?

@dmcgowan

This comment has been minimized.

Copy link
Member Author

commented Feb 15, 2019

Is this planned to be merged before Docker v19.03?

That is desirable but unrealistic. I want to complete as many or all of the major things by that time frame, but until all the functionality is complete it is too hard to pin down an exact date for "stable enough to merge".

It is very important we get the design parts mentioned in "highlights" correct so we aren't in a position where we need to correct storage code later (adding more support permutations and instability). Please comment on the design and help wherever possible will certainly help stabilize this change.

const (
// LabelImageID refers to the image ID used by Docker
// Deprecate this to support multi-arch images
LabelImageID = "docker.io/image.id"

This comment has been minimized.

Copy link
@AkihiroSuda

AkihiroSuda Feb 16, 2019

Member

nit: mobyproject.org? docker.com?

This comment has been minimized.

Copy link
@thaJeztah

thaJeztah Mar 19, 2019

Member

If these can conflict with user-provided labels; reverse-dns notation was originally marked as reserved; https://docs.docker.com/config/labels-custom-metadata/#key-format-recommendations

The com.docker.*, io.docker.*, and org.dockerproject.* namespaces are reserved by Docker for internal use.

@cpuguy83

This comment has been minimized.

Copy link
Contributor

commented Mar 2, 2019

Pushed a rebase.

@dmcgowan dmcgowan force-pushed the dmcgowan:containerd-integration branch 4 times, most recently from 3018ba4 to a2ee367 Mar 4, 2019

@cpuguy83 cpuguy83 force-pushed the dmcgowan:containerd-integration branch from ee7d1b3 to 1888615 Mar 6, 2019

@dmcgowan dmcgowan force-pushed the dmcgowan:containerd-integration branch from 1888615 to b7480ac Mar 11, 2019

@dmcgowan dmcgowan force-pushed the dmcgowan:containerd-integration branch from b7db406 to f3dacb4 Mar 13, 2019

@GordonTheTurtle GordonTheTurtle removed the dco/no label Mar 13, 2019

@moby moby deleted a comment from GordonTheTurtle Mar 13, 2019

@dmcgowan dmcgowan force-pushed the dmcgowan:containerd-integration branch from f3dacb4 to 9d443a0 Mar 13, 2019

@GordonTheTurtle

This comment has been minimized.

Copy link

commented Mar 14, 2019

Please sign your commits following these rules:
https://github.com/moby/moby/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:

$ git clone -b "containerd-integration" git@github.com:dmcgowan/docker.git somewhere
$ cd somewhere
$ git rebase -i HEAD~842359062744
editor opens
change each 'pick' to 'edit'
save the file and quit
$ git commit --amend -s --no-edit
$ git rebase --continue # and repeat the amend for each commit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.

dmcgowan and others added some commits Mar 20, 2019

Show digest tags from dangling images
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Add size to docker images
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Improve platform matching for containers and pull
By default match a single architecture platform for
pull. Use container platform for resolving layerstore.
Use container driver name when available to resolve
layerstore.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Remove deprecated cache
Use layer cache when necessary or containerd client directly

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Fix docker images filters
Support dangling only and reference lookups

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Optimize image list to skip size on skipped images
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Fix image lookup and listing
Ensure ambiguous reference is not mistakenly returned.
Cleanup TODOs in listing and resolve image

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Fix unit tests and lint
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
use containerd client to create image during build
Signed-off-by: Anda Xu <axu@knox.edu>
Fix CI linting
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Revert image package changes
The entire package is now deprecated, revert temporary
changes.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
add already exists image check during pull
Signed-off-by: Anda Xu <axu@knox.edu>
Support updating existing image after pull
Fix build

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Fix swagger nonsense
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Pass test debug flag from environment
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Add load implementation using containerd import
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Update make emptyfs to use supported load format
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Update output to match integration tests
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
image: restore(): prevent panic
This is to avoid nil pointer dereference.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Preliminary image store migration support
This migrates image store to containerd.

To test:
 # sudo DOCKER_MIGRATE_IMAGE_STORE=da ./bundles/dynbinary-daemon/dockerd-dev

To re-test:
 # ctr -n moby image ls -q | xargs ctr -n moby image rm
 # ctr -n moby content ls -q | xargs ctr -n moby content rm

TODO:
 - find a better place for this code
 - maybe some tests?

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
clean up code and add comments to image_builder
Signed-off-by: Anda Xu <axu@knox.edu>
resolve runtime image properly for both builder and end user
fix already exists error during image tagging

Signed-off-by: Anda Xu <axu@knox.edu>
Move migration to separate function
Remove image store instantiation from image service

Signed-off-by: Derek McGowan <derek@mcgstyle.net>

@dmcgowan dmcgowan force-pushed the dmcgowan:containerd-integration branch 2 times, most recently from 7530b9c to 9f7f374 Apr 8, 2019

Fix validation
Signed-off-by: Derek McGowan <derek@mcgstyle.net>

@dmcgowan dmcgowan force-pushed the dmcgowan:containerd-integration branch from 9f7f374 to 713282b Apr 8, 2019

dmcgowan added some commits Apr 10, 2019

Update migration to use layer stores
Move logic to image service and merge with cache loading.
Ignore not exists errors on migration
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.