Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] image store migration #38925

Open
wants to merge 56 commits into
base: master
from

Conversation

Projects
None yet
6 participants
@kolyshkin
Copy link
Contributor

kolyshkin commented Mar 22, 2019

This PR builds on top of (and currently includes) #38738.

Provide a way to migrate image store to containerd.

To test (assuming you use ./hack/make.sh dynbinary for building):

$ sudo DOCKER_MIGRATE_IMAGE_STORE=да ./bundles/dynbinary-daemon/dockerd-dev

To prepare for re-test:

# ctr -n moby image ls -q | xargs ctr -n moby image rm
# ctr -n moby content ls -q | xargs ctr -n moby content rm

TODO:

  • find a better place for this code
  • make the code less ugly
  • maybe even write some tests?
  • uncomment src objects removal?

@dmcg PTAL

dmcgowan and others added some commits Oct 16, 2018

Use containerd client to pull images
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Use containerd to list images
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Use containerd to create tags
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Add image cache
Use cache for delete and listing images

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Add image tagging by id and reference
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Add image deletion
Fix conflict and update removal to use cache and not stores.
Layer removal is still missing.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Add support for layers
Unpack and delete layers

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Add support for inspect
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Update image history to use containerd
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Add support for docker run
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Update image service functions to use containerd
Updates info functions to query the containerd image cache.
Updates related call stack to pass down context.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
display docker pull progress for containerd integration
Signed-off-by: Anda Xu <axu@knox.edu>
Add support for commit
Fix dangling image management to not rely on reference parsing

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Add parent label to committed images
Fix layer references being taken on startup

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Update events handler to use containerd
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Add push support through containerd
Uses containerd client to push.
Temporarily resolves current platform on push to match current behavior.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Update images to use manifest as image ID
Lookup images without using caching

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
unpack layers simultaneously
Signed-off-by: Anda Xu <axu@knox.edu>
Add layer label to content
Adds equivalent label as the snapshotter gc label

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Resolve runtime image
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
unblock layer extracting to a separate thread
Signed-off-by: Anda Xu <axu@knox.edu>
Move layer creation to daemon package
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Support for multiple layer stores
Make layer stores multiplatform and allow configuring
multiple ordered by preference.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Update pull to use error groups
Support for multiple unpacks

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Lookup images by digest regexp
Filter pull artifacts by platform while pulling
all manifests and configs.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Use content label for inspecting layer store
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Update pull to add canonical tag
Update tag to avoid deprecate cache

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Add test framework which sets up containerd
Add initial test for listing images

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Update rmi to use containerd gc
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Add delete image unit tests
Signed-off-by: Derek McGowan <derek@mcgstyle.net>

dmcgowan and others added some commits Mar 20, 2019

Update history to not rely on deprecated cache
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Show digest tags from dangling images
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Add size to docker images
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Improve platform matching for containers and pull
By default match a single architecture platform for
pull. Use container platform for resolving layerstore.
Use container driver name when available to resolve
layerstore.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Remove deprecated cache
Use layer cache when necessary or containerd client directly

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Fix docker images filters
Support dangling only and reference lookups

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Optimize image list to skip size on skipped images
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Fix image lookup and listing
Ensure ambiguous reference is not mistakenly returned.
Cleanup TODOs in listing and resolve image

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Fix unit tests and lint
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
use containerd client to create image during build
Signed-off-by: Anda Xu <axu@knox.edu>
Fix CI linting
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Revert image package changes
The entire package is now deprecated, revert temporary
changes.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Mar 22, 2019

DOCKER_MIGRATE_IMAGE_STORE=да

🤔😅🇷🇺 does it work in other languages?

AntaresS and others added some commits Mar 22, 2019

add already exists image check during pull
Signed-off-by: Anda Xu <axu@knox.edu>
Support updating existing image after pull
Fix build

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Fix swagger nonsense
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Pass test debug flag from environment
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Add load implementation using containerd import
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Update make emptyfs to use supported load format
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Move python tests after integration tests
Run more internal tests of Docker before testing with external libraries.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Update output to match integration tests
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
image: restore(): prevent panic
This is to avoid nil pointer dereference.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Preliminary image store migration support
This migrates image store to containerd.

To test:
 # sudo DOCKER_MIGRATE_IMAGE_STORE=da ./bundles/dynbinary-daemon/dockerd-dev

To re-test:
 # ctr -n moby image ls -q | xargs ctr -n moby image rm
 # ctr -n moby content ls -q | xargs ctr -n moby content rm

TODO:
 - find a better place for this code
 - maybe some tests?

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

@kolyshkin kolyshkin force-pushed the kolyshkin:mig branch from bab0eb6 to 33ce0fa Mar 27, 2019

@kolyshkin kolyshkin requested a review from tianon as a code owner Mar 27, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.