New Data Management commands #26108

Merged
merged 10 commits into from Oct 1, 2016

Conversation

@mlaventure
Contributor

mlaventure commented Aug 29, 2016

This PR introduce a few new commands to help facilitate visualising how much space the docker daemon data is taking on disk and allowing for easily cleaning up "unneeded" excess.

  • docker system df presents a summary of the space currently used by different docker objects:
TYPE                TOTAL               ACTIVE              SIZE                UNUSED
Images              5                   2                   16.43 MB            11.63 MB (70%)
Containers          2                   0                   212 B               212 B (100%)
Local Volumes             2                   1                   36 B                0 B (0%)
  • docker system df -v expends on the previous with more details:
Images space usage:

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE                SHARED SIZE         UNIQUE SIZE         CONTAINERS
my-curl             latest              b2789dd875bf        6 minutes ago       11 MB               11 MB               5 B                 0
my-jq               latest              ae67841be6d0        6 minutes ago       9.623 MB            8.991 MB            632.1 kB            0
<none>              <none>              a0971c4015c1        6 minutes ago       11 MB               11 MB               0 B                 0
alpine              latest              4e38e38c8ce0        9 weeks ago         4.799 MB            0 B                 4.799 MB            1
alpine              3.3                 47cf20d8c26c        9 weeks ago         4.797 MB            4.797 MB            0 B                 1

Containers space usage:

CONTAINER ID        IMAGE               COMMAND             LOCAL VOLUMES       SIZE                CREATED             STATUS                      NAMES
4a7f7eebae0f        alpine:latest       "sh"                1                   0 B                 16 minutes ago      Exited (0) 5 minutes ago    hopeful_yalow
f98f9c2aa1ea        alpine:3.3          "sh"                1                   212 B               16 minutes ago      Exited (0) 48 seconds ago   anon-vol

Local Volumes space usage:

NAME                                                               LINKS               SIZE
07c7bdf3e34ab76d921894c2b834f073721fccfbbcba792aa7648e3a7a664c2e   2                   36 B
my-named-vol                                                       0                   0 B
  • docker system prune will delete ALL unused data (i.e. In order: containers stopped, volumes without containers and images with no containers):
WARNING! This will remove:
    - all stopped containers
    - all volumes not used by at least one container
    - all images without at least one container associated to them
Are you sure you want to continue? [y/N] y
Deleted Containers:
4a7f7eebae0f63178aff7eb0aa39cd3f0627a203ab2df258c1a00b456cf20063
f98f9c2aa1eaf727e4ec9c0283bc7d4aa4762fbdba7f26191f26c97f64090360

Deleted Volumes:
07c7bdf3e34ab76d921894c2b834f073721fccfbbcba792aa7648e3a7a664c2e
my-named-vol

Deleted Images:
untagged: alpine:latest
untagged: alpine@sha256:3dcdb92d7432d56604d4545cbd324b14e647b313626d99b889d0626de158f73a
deleted: sha256:4e38e38c8ce0b8d9041a9c4fefe786631d1416225e13b0bfe8cfa2321aec4bba
deleted: sha256:4fe15f8d0ae69e169824f25f1d4da3015a48feeeeebb265cd2e328e15c6a869f
untagged: alpine:3.3
untagged: alpine@sha256:4fa633f4feff6a8f02acfc7424efd5cb3e76686ed3218abf4ca0fa4a2a358423
untagged: my-jq:latest
deleted: sha256:ae67841be6d008a374eff7c2a974cde3934ffe9536a7dc7ce589585eddd83aff
deleted: sha256:34f6f1261650bc341eb122313372adc4512b4fceddc2a7ecbb84f0958ce5ad65
deleted: sha256:cf4194e8d8db1cb2d117df33f2c75c0369c3a26d96725efb978cc69e046b87e7
untagged: my-curl:latest
deleted: sha256:b2789dd875bf427de7f9f6ae001940073b3201409b14aba7e5db71f408b8569e
deleted: sha256:96daac0cb203226438989926fc34dd024f365a9a8616b93e168d303cfe4cb5e9
deleted: sha256:5cbd97a14241c9cd83250d6b6fc0649833c4a3e84099b968dd4ba403e609945e
deleted: sha256:a0971c4015c1e898c60bf95781c6730a05b5d8a2ae6827f53837e6c9d38efdec
deleted: sha256:d8359ca3b681cc5396a4e790088441673ed3ce90ebc04de388bfcd31a0716b06
deleted: sha256:83fc9ba8fb70e1da31dfcc3c88d093831dbd4be38b34af998df37e8ac538260c
deleted: sha256:ae7041a4cc625a9c8e6955452f7afe602b401f662671cea3613f08f3d9343b35
deleted: sha256:35e0f43a37755b832f0bbea91a2360b025ee351d7309dae0d9737bc96b6d0809
deleted: sha256:0af941dd29f00e4510195dd00b19671bc591e29d1495630e7e0f7c44c1e6a8c0
deleted: sha256:9fc896fc2013da84f84e45b3096053eb084417b42e6b35ea0cce5a3529705eac
deleted: sha256:47cf20d8c26c46fff71be614d9f54997edacfe8d46d51769706e5aba94b16f2b
deleted: sha256:2c675ee9ed53425e31a13e3390bf3f539bf8637000e4bcfbb85ee03ef4d910a1

Total reclaimed space: 17 MB
  • docker container prune will delete all stopped containers:
WARNING! This will remove all stopped containers.
Are you sure you want to continue? [y/N] y
Deleted Containers:
4a7f7eebae0f63178aff7eb0aa39cd3f0627a203ab2df258c1a00b456cf20063
f98f9c2aa1eaf727e4ec9c0283bc7d4aa4762fbdba7f26191f26c97f64090360

Total reclaimed space: 212 B
  • docker volume prune will delete all volume without associated containers:
WARNING! This will remove all volumes not used by at least one container.
Are you sure you want to continue? [y/N] y
Deleted Volumes:
07c7bdf3e34ab76d921894c2b834f073721fccfbbcba792aa7648e3a7a664c2e
my-named-vol

Total reclaimed space: 36 B
  • docker image prune will delete all image without associated containers:
WARNING! This will remove all images without at least one container associated to them.
Are you sure you want to continue? [y/N] y
Deleted Images:
untagged: alpine:latest
untagged: alpine@sha256:3dcdb92d7432d56604d4545cbd324b14e647b313626d99b889d0626de158f73a
deleted: sha256:4e38e38c8ce0b8d9041a9c4fefe786631d1416225e13b0bfe8cfa2321aec4bba
deleted: sha256:4fe15f8d0ae69e169824f25f1d4da3015a48feeeeebb265cd2e328e15c6a869f
untagged: alpine:3.3
untagged: alpine@sha256:4fa633f4feff6a8f02acfc7424efd5cb3e76686ed3218abf4ca0fa4a2a358423
untagged: my-jq:latest
deleted: sha256:ae67841be6d008a374eff7c2a974cde3934ffe9536a7dc7ce589585eddd83aff
deleted: sha256:34f6f1261650bc341eb122313372adc4512b4fceddc2a7ecbb84f0958ce5ad65
deleted: sha256:cf4194e8d8db1cb2d117df33f2c75c0369c3a26d96725efb978cc69e046b87e7
untagged: my-curl:latest
deleted: sha256:b2789dd875bf427de7f9f6ae001940073b3201409b14aba7e5db71f408b8569e
deleted: sha256:96daac0cb203226438989926fc34dd024f365a9a8616b93e168d303cfe4cb5e9
deleted: sha256:5cbd97a14241c9cd83250d6b6fc0649833c4a3e84099b968dd4ba403e609945e
deleted: sha256:a0971c4015c1e898c60bf95781c6730a05b5d8a2ae6827f53837e6c9d38efdec
deleted: sha256:d8359ca3b681cc5396a4e790088441673ed3ce90ebc04de388bfcd31a0716b06
deleted: sha256:83fc9ba8fb70e1da31dfcc3c88d093831dbd4be38b34af998df37e8ac538260c
deleted: sha256:ae7041a4cc625a9c8e6955452f7afe602b401f662671cea3613f08f3d9343b35
deleted: sha256:35e0f43a37755b832f0bbea91a2360b025ee351d7309dae0d9737bc96b6d0809
deleted: sha256:0af941dd29f00e4510195dd00b19671bc591e29d1495630e7e0f7c44c1e6a8c0
deleted: sha256:9fc896fc2013da84f84e45b3096053eb084417b42e6b35ea0cce5a3529705eac
deleted: sha256:47cf20d8c26c46fff71be614d9f54997edacfe8d46d51769706e5aba94b16f2b
deleted: sha256:2c675ee9ed53425e31a13e3390bf3f539bf8637000e4bcfbb85ee03ef4d910a1

Total reclaimed space: 16.43 MB

Things left to do:

  • Add the documentation for the new commands
  • Proper vendoring

Relates to #18601
Fixes #9054.
Closes #22871.
Closes #15331
Depends on #26025 (merged)

ping @dnephin @tonistiigi @icecrime

@duglin

This comment has been minimized.

Show comment
Hide comment
@duglin

duglin Aug 29, 2016

Contributor

lots of people have asked for this type of thing! nice!

Contributor

duglin commented Aug 29, 2016

lots of people have asked for this type of thing! nice!

@aaronlehmann

This comment has been minimized.

Show comment
Hide comment
@aaronlehmann

aaronlehmann Aug 29, 2016

Contributor

Very nice.

I'm not sure if it's within the scope of this PR, but it would be cool to have some kind of tree view of which layers are used by which images. That would show how to reclaim space used by large layers.

Contributor

aaronlehmann commented Aug 29, 2016

Very nice.

I'm not sure if it's within the scope of this PR, but it would be cool to have some kind of tree view of which layers are used by which images. That would show how to reclaim space used by large layers.

@crosbymichael

This comment has been minimized.

Show comment
Hide comment
@crosbymichael

crosbymichael Aug 29, 2016

Contributor

@aaronlehmann no, as a maintainer you are to never mention some type of "tree view" again ;)

Contributor

crosbymichael commented Aug 29, 2016

@aaronlehmann no, as a maintainer you are to never mention some type of "tree view" again ;)

@duglin

This comment has been minimized.

Show comment
Hide comment
@duglin

duglin Aug 29, 2016

Contributor

LOL i was wondering if someone was going to correct this egregious mistake :-)

Contributor

duglin commented Aug 29, 2016

LOL i was wondering if someone was going to correct this egregious mistake :-)

@dperny

This comment has been minimized.

Show comment
Hide comment
@dperny

dperny Aug 29, 2016

Contributor

Might be a good idea to add an -f/--force option to the prune commands, so that you don't have to pipe yes to them if you're using them in a script, but I see this is WIP so maybe you are already planning on doing this.

Contributor

dperny commented Aug 29, 2016

Might be a good idea to add an -f/--force option to the prune commands, so that you don't have to pipe yes to them if you're using them in a script, but I see this is WIP so maybe you are already planning on doing this.

@icecrime

This comment has been minimized.

Show comment
Hide comment
@icecrime

icecrime Aug 29, 2016

Contributor

Design LGTM 👍

I have to say that I'm not a big fan of the output of docker system df -v (this kind of "list of tables" looks weird to me), but I don't have a better proposal.

Contributor

icecrime commented Aug 29, 2016

Design LGTM 👍

I have to say that I'm not a big fan of the output of docker system df -v (this kind of "list of tables" looks weird to me), but I don't have a better proposal.

@tonistiigi

This comment has been minimized.

Show comment
Hide comment
@tonistiigi

tonistiigi Aug 29, 2016

Member

So why not start it as a client side implementation first? As the /prune has an undefined result because we don't have object versioning it has limited uses for someone trying to use it through API in their scripts. I imagine in the future we want to add confirmation and -i then this would work much more like our current API that has deletion for single objects.

Member

tonistiigi commented Aug 29, 2016

So why not start it as a client side implementation first? As the /prune has an undefined result because we don't have object versioning it has limited uses for someone trying to use it through API in their scripts. I imagine in the future we want to add confirmation and -i then this would work much more like our current API that has deletion for single objects.

@vieux

This comment has been minimized.

Show comment
Hide comment
@vieux

vieux Aug 29, 2016

Collaborator

nice, look good.

nit:
docker system df -v displays title like <TYPE> space usage:
you can see TYPE in docker system df

It is consistent for containers and images but not for volumes.

docker system df says Volumes
and docker system df -v says Local Volumes

Collaborator

vieux commented Aug 29, 2016

nice, look good.

nit:
docker system df -v displays title like <TYPE> space usage:
you can see TYPE in docker system df

It is consistent for containers and images but not for volumes.

docker system df says Volumes
and docker system df -v says Local Volumes

@yosifkit

This comment has been minimized.

Show comment
Hide comment
@yosifkit

yosifkit Aug 29, 2016

Contributor

Just to add some perspective on how to do the same prune sets in bash.

$ # caveat that you get warnings on things that can't be removed

$ # remove all stopped containers
$ docker ps -aq | xargs --no-run-if-empty docker rm

$ # remove all unused volumes
$ docker volume ls -q | xargs --no-run-if-empty docker volume rm
$ # remove local volumes  
$ docker volume ls | awk '/^local/ { print $2 }' | xargs --no-run-if-empty docker volume rm

$ # delete untagged images
$ docker images --filter dangling=true -q | xargs --no-run-if-empty docker rmi

$ # delete and untag every image that is not a container
$ # a little more heinous since "<none>" is the repo/tag for dangling images
$ docker images | awk 'NR>1 { if ($1 == "<none>") print $3; else print $1":"$2 }' | sort | xargs --no-run-if-empty docker rmi
$ # sort is not necessary but is nice if you add a -tn1 to xargs so you can see each rm line

What about the volumes of stopped containers? Shouldn't the docker container prune have a -v flag to also remove the associated volumes?

Should it be docker container ...? We don't have any other command that uses "container", since many commands are assumed to be working on a container (attach, commit, cp, diff, exec, start, stats, stop, kill, rm, etc).

Should probably change docker image prune -> docker images prune. Maybe even start deprecating images to be images ls or image ls to be consistent with docker volume ls (or fix docker volume consistency).

It would also be help to have some sort of --filter to specifically exclude certain images, volumes, or containers. Without filtering, I feel like many people may resort to similar one-liners so that they can throw a grep into the pipe.

Contributor

yosifkit commented Aug 29, 2016

Just to add some perspective on how to do the same prune sets in bash.

$ # caveat that you get warnings on things that can't be removed

$ # remove all stopped containers
$ docker ps -aq | xargs --no-run-if-empty docker rm

$ # remove all unused volumes
$ docker volume ls -q | xargs --no-run-if-empty docker volume rm
$ # remove local volumes  
$ docker volume ls | awk '/^local/ { print $2 }' | xargs --no-run-if-empty docker volume rm

$ # delete untagged images
$ docker images --filter dangling=true -q | xargs --no-run-if-empty docker rmi

$ # delete and untag every image that is not a container
$ # a little more heinous since "<none>" is the repo/tag for dangling images
$ docker images | awk 'NR>1 { if ($1 == "<none>") print $3; else print $1":"$2 }' | sort | xargs --no-run-if-empty docker rmi
$ # sort is not necessary but is nice if you add a -tn1 to xargs so you can see each rm line

What about the volumes of stopped containers? Shouldn't the docker container prune have a -v flag to also remove the associated volumes?

Should it be docker container ...? We don't have any other command that uses "container", since many commands are assumed to be working on a container (attach, commit, cp, diff, exec, start, stats, stop, kill, rm, etc).

Should probably change docker image prune -> docker images prune. Maybe even start deprecating images to be images ls or image ls to be consistent with docker volume ls (or fix docker volume consistency).

It would also be help to have some sort of --filter to specifically exclude certain images, volumes, or containers. Without filtering, I feel like many people may resort to similar one-liners so that they can throw a grep into the pipe.

@icecrime

This comment has been minimized.

Show comment
Hide comment
@icecrime

icecrime Aug 29, 2016

Contributor

What about the volumes of stopped containers? Shouldn't the docker container prune have a -v flag to also remove the associated volumes?

I think it's more natural if docker container prune only applies to containers. Because docker prune does container first, it means that volumes referenced by stopped containers will get removed too.

Should it be docker container ...? We don't have any other command that uses "container", since many commands are assumed to be working on a container (attach, commit, cp, diff, exec, start, stats, stop, kill, rm, etc).
Should probably change docker image prune -> docker images prune. Maybe even start deprecating images to be images ls or image ls to be consistent with docker volume ls (or fix docker volume consistency).

This PR assumes that #26025 will be merged: you should look at that one for more context 😉

It would also be help to have some sort of --filter to specifically exclude certain images, volumes, or containers. Without filtering, I feel like many people may resort to similar one-liners so that they can throw a grep into the pipe.

I agree, I'm pretty sure this will come in pretty fast once this one gets merged. Do you agree it's a solid starting point nevertheless, or do you feel it's missing something to be useful?

Contributor

icecrime commented Aug 29, 2016

What about the volumes of stopped containers? Shouldn't the docker container prune have a -v flag to also remove the associated volumes?

I think it's more natural if docker container prune only applies to containers. Because docker prune does container first, it means that volumes referenced by stopped containers will get removed too.

Should it be docker container ...? We don't have any other command that uses "container", since many commands are assumed to be working on a container (attach, commit, cp, diff, exec, start, stats, stop, kill, rm, etc).
Should probably change docker image prune -> docker images prune. Maybe even start deprecating images to be images ls or image ls to be consistent with docker volume ls (or fix docker volume consistency).

This PR assumes that #26025 will be merged: you should look at that one for more context 😉

It would also be help to have some sort of --filter to specifically exclude certain images, volumes, or containers. Without filtering, I feel like many people may resort to similar one-liners so that they can throw a grep into the pipe.

I agree, I'm pretty sure this will come in pretty fast once this one gets merged. Do you agree it's a solid starting point nevertheless, or do you feel it's missing something to be useful?

@yosifkit

This comment has been minimized.

Show comment
Hide comment
@yosifkit

yosifkit Aug 29, 2016

Contributor

Thanks @icecrime!

Do you agree it's a solid starting point nevertheless, or do you feel it's missing something to be useful?

I am still on the fence as to whether this is strictly necessary, but I do see it being used.

I am not a fan of the y/n prompt; the only place I can find this currently within docker is in api/client/plugin/install.go.

I feel like docker image prune should default to the more "safe" behavior of just dangling images with a flag of something like --aggressive (resembling git gc --prune=now --aggressive) to get rid of every image that is not a running container. This could let us remove the y/n prompt. Though I am not sure what the non-aggressive version would be for containers and volumes.

Edit: a non-aggressive prune on containers could only delete stopped containers that do not have a volume. Or maybe only successful exit codes.

Contributor

yosifkit commented Aug 29, 2016

Thanks @icecrime!

Do you agree it's a solid starting point nevertheless, or do you feel it's missing something to be useful?

I am still on the fence as to whether this is strictly necessary, but I do see it being used.

I am not a fan of the y/n prompt; the only place I can find this currently within docker is in api/client/plugin/install.go.

I feel like docker image prune should default to the more "safe" behavior of just dangling images with a flag of something like --aggressive (resembling git gc --prune=now --aggressive) to get rid of every image that is not a running container. This could let us remove the y/n prompt. Though I am not sure what the non-aggressive version would be for containers and volumes.

Edit: a non-aggressive prune on containers could only delete stopped containers that do not have a volume. Or maybe only successful exit codes.

@icecrime

This comment has been minimized.

Show comment
Hide comment
@icecrime

icecrime Aug 30, 2016

Contributor

Ha, I didn't realize that there was indeed a significant difference between "dangling" and "not referenced by any container" in the context of images.

  • You're right that docker image prune removing dangling images is much less dangerous.
  • Now that I'm looking at it, I see that "dangling" in the context of volumes does mean "not referenced by any container"... Another one for consistency 🍺

WDYT @mlaventure?

Contributor

icecrime commented Aug 30, 2016

Ha, I didn't realize that there was indeed a significant difference between "dangling" and "not referenced by any container" in the context of images.

  • You're right that docker image prune removing dangling images is much less dangerous.
  • Now that I'm looking at it, I see that "dangling" in the context of volumes does mean "not referenced by any container"... Another one for consistency 🍺

WDYT @mlaventure?

@mlaventure

This comment has been minimized.

Show comment
Hide comment
@mlaventure

mlaventure Aug 30, 2016

Contributor

I don't mind having it only removing dangling image by default. Finding the
right flag name is more of an issue. --aggressive sounds scary, but again
maybe that's a good thing given what the command does.

On Mon, Aug 29, 2016, 5:08 PM Arnaud Porterie notifications@github.com
wrote:

Ha, I didn't realize that there was indeed a significant difference
between "dangling" and "not referenced by any container" in the context of
images.

WDYT @mlaventure https://github.com/mlaventure?


You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
#26108 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/APUifoQGN4la4JH-cDihpfgfvmzlJmVcks5qk3RcgaJpZM4Jvr89
.

Contributor

mlaventure commented Aug 30, 2016

I don't mind having it only removing dangling image by default. Finding the
right flag name is more of an issue. --aggressive sounds scary, but again
maybe that's a good thing given what the command does.

On Mon, Aug 29, 2016, 5:08 PM Arnaud Porterie notifications@github.com
wrote:

Ha, I didn't realize that there was indeed a significant difference
between "dangling" and "not referenced by any container" in the context of
images.

WDYT @mlaventure https://github.com/mlaventure?


You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
#26108 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/APUifoQGN4la4JH-cDihpfgfvmzlJmVcks5qk3RcgaJpZM4Jvr89
.

@AkihiroSuda

This comment has been minimized.

Show comment
Hide comment
@AkihiroSuda

AkihiroSuda Aug 30, 2016

Member

Maybe prune should keep "young" resources (e.g. images created in last 5 minutes) by default (or at least, how to keep them using --filter should be documented).

People may face difficulty in running docker build && docker run when the image just they created is pruned by a cron job that executes docker system prune.

Member

AkihiroSuda commented Aug 30, 2016

Maybe prune should keep "young" resources (e.g. images created in last 5 minutes) by default (or at least, how to keep them using --filter should be documented).

People may face difficulty in running docker build && docker run when the image just they created is pruned by a cron job that executes docker system prune.

@AkihiroSuda

This comment has been minimized.

Show comment
Hide comment
@AkihiroSuda

AkihiroSuda Aug 30, 2016

Member

Maybe it is useful if there is docker network prune as well. (Then df should be renamed?)

Member

AkihiroSuda commented Aug 30, 2016

Maybe it is useful if there is docker network prune as well. (Then df should be renamed?)

@icecrime

This comment has been minimized.

Show comment
Hide comment
@icecrime

icecrime Aug 30, 2016

Contributor

I'm sure filtering will be a must have, but we need a starting point 😞 This looks to me as a PR we can build upon.

Regarding networking, that's a good point. It would indeed make sense to implement docker network prune, but I don't think it would need to appear in docker system df if we want to make this command strictly about disk space.

Contributor

icecrime commented Aug 30, 2016

I'm sure filtering will be a must have, but we need a starting point 😞 This looks to me as a PR we can build upon.

Regarding networking, that's a good point. It would indeed make sense to implement docker network prune, but I don't think it would need to appear in docker system df if we want to make this command strictly about disk space.

@mlaventure

This comment has been minimized.

Show comment
Hide comment
@mlaventure

mlaventure Aug 31, 2016

Contributor

Added:

  • documentation
  • -f flag to remove the prompt
  • made image prune defaulting to dangling only unless -a is provided.
Contributor

mlaventure commented Aug 31, 2016

Added:

  • documentation
  • -f flag to remove the prompt
  • made image prune defaulting to dangling only unless -a is provided.

@thaJeztah thaJeztah referenced this pull request Sep 14, 2016

Closed

Docker disk usage #12265

@dnephin dnephin referenced this pull request Sep 19, 2016

Merged

Create system subcommand #26716

@icecrime icecrime changed the title from [WIP] New Data Management commands to New Data Management commands Sep 19, 2016

@mlaventure

This comment has been minimized.

Show comment
Hide comment
@mlaventure

mlaventure Sep 21, 2016

Contributor

@tonistiigi correctly pointed that adding proper confirmation (i.e. print a list of everything that will try to get removed to the user before processing) with the current state of the API would be quite troublesome later on.

As a result, I'm going to have to move the prune logic into the client side.

Contributor

mlaventure commented Sep 21, 2016

@tonistiigi correctly pointed that adding proper confirmation (i.e. print a list of everything that will try to get removed to the user before processing) with the current state of the API would be quite troublesome later on.

As a result, I'm going to have to move the prune logic into the client side.

@mlaventure mlaventure changed the title from New Data Management commands to WIP: New Data Management commands Sep 21, 2016

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Sep 21, 2016

Member

Possibly "pinning" / "locking" containers/images/volumes becomes more relevant with this (see #7017

Member

thaJeztah commented Sep 21, 2016

Possibly "pinning" / "locking" containers/images/volumes becomes more relevant with this (see #7017

@dnephin

I think server-side prune can work, as long as:

  • we design it to accept lots of different filters in the future
  • we make it clear in the --dry-run option that it's only a "preview", and results may change if the properties of the objects change between command invocations
api/types/types.go
+ PruneVolumes bool
+ PruneImages bool
+ DanglingImagesOnly bool
+}

This comment has been minimized.

@dnephin

dnephin Sep 21, 2016

Member

Instead of having just a single endpoint for prune, I think it might be better to have one endpoint per type (matching the cli).

The request data is going to expand a lot with different filtering options, and I think it's easier to represent that in a sane way by having a request type for each prune-able object.

@dnephin

dnephin Sep 21, 2016

Member

Instead of having just a single endpoint for prune, I think it might be better to have one endpoint per type (matching the cli).

The request data is going to expand a lot with different filtering options, and I think it's easier to represent that in a sane way by having a request type for each prune-able object.

This comment has been minimized.

@mlaventure

mlaventure Sep 22, 2016

Contributor

Will make the change

@mlaventure

mlaventure Sep 22, 2016

Contributor

Will make the change

@mlaventure mlaventure changed the title from WIP: New Data Management commands to New Data Management commands Sep 23, 2016

@mlaventure

This comment has been minimized.

Show comment
Hide comment
@mlaventure

mlaventure Sep 23, 2016

Contributor

ping @dnephin, I made the split

Contributor

mlaventure commented Sep 23, 2016

ping @dnephin, I made the split

@dnephin

This comment has been minimized.

Show comment
Hide comment
@dnephin

dnephin Sep 23, 2016

Member

Thanks! LGTM

Member

dnephin commented Sep 23, 2016

Thanks! LGTM

@mlaventure

This comment has been minimized.

Show comment
Hide comment
+ }
+
+ flags := cmd.Flags()
+ flags.BoolVarP(&opts.force, "force", "f", false, "Do not prompt for confirmation")

This comment has been minimized.

@AkihiroSuda

AkihiroSuda Sep 23, 2016

Member

"Do not show a prompt"?

@AkihiroSuda

AkihiroSuda Sep 23, 2016

Member

"Do not show a prompt"?

This comment has been minimized.

@dnephin

dnephin Sep 27, 2016

Member

I like the current text better

@dnephin

dnephin Sep 27, 2016

Member

I like the current text better

+ --help Print usage
+```
+
+Delete all images dangling images. If `-a` is specified, will also remove all images not referenced by any container.

This comment has been minimized.

@AkihiroSuda

AkihiroSuda Sep 23, 2016

Member

"Delete all dangling images"?

@AkihiroSuda

AkihiroSuda Sep 23, 2016

Member

"Delete all dangling images"?

This comment has been minimized.

@mlaventure

mlaventure Sep 23, 2016

Contributor

Thanks

@mlaventure

mlaventure Sep 23, 2016

Contributor

Thanks

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Sep 30, 2016

Member

ping @docker/docs-maintainers PTAL if this needs changes in other parts of the documentation

Member

thaJeztah commented Sep 30, 2016

ping @docker/docs-maintainers PTAL if this needs changes in other parts of the documentation

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Sep 30, 2016

Member

I'll go ahead and move this to "merge"; documentation team is busy with the migration still, and we can do a follow up if needed

Member

thaJeztah commented Sep 30, 2016

I'll go ahead and move this to "merge"; documentation team is busy with the migration still, and we can do a follow up if needed

@icecrime

This comment has been minimized.

Show comment
Hide comment
@icecrime

icecrime Oct 1, 2016

Contributor

Windows failure seems unrelated:

01:19:43 FAIL: docker_cli_run_test.go:1855: DockerSuite.TestRunInteractiveWithRestartPolicy
01:19:43 
01:19:43 docker_cli_run_test.go:1868:
01:19:43     c.Assert(result, icmd.Matches, icmd.Expected{ExitCode: 11})
01:19:43 ... result *cmd.Result = &cmd.Result{Cmd:(*exec.Cmd)(0xc0423846e0), ExitCode:0, Error:error(nil), Timeout:true, outBuffer:(*cmd.lockedBuffer)(0xc0439e9d40), errBuffer:(*cmd.lockedBuffer)(0xc0439e9dd0)} ("\nCommand: d:\\CI\\CI-cc2ee71\\binary\\docker.exe run -i --name test-inter-restart --restart=always busybox sh\nExitCode: 0 (timeout), Error: %!s(<nil>)\nStdout: \nStderr: \n")
01:19:43 ... expected cmd.Expected = cmd.Expected{ExitCode:11, Timeout:false, Error:"", Out:"", Err:""}
01:19:43 ... 
01:19:43 Command: d:\CI\CI-cc2ee71\binary\docker.exe run -i --name test-inter-restart --restart=always busybox sh
01:19:43 ExitCode: 0 (timeout), Error: %!s(<nil>)
01:19:43 Stdout: 
01:19:43 Stderr: 
01:19:43 
01:19:43 Failures:
01:19:43 ExitCode was 0 expected 11
01:19:43 Expected command to finish, but it hit the timeout

Merging: thanks @mlaventure, and thanks to all the reviewers ❤️

Contributor

icecrime commented Oct 1, 2016

Windows failure seems unrelated:

01:19:43 FAIL: docker_cli_run_test.go:1855: DockerSuite.TestRunInteractiveWithRestartPolicy
01:19:43 
01:19:43 docker_cli_run_test.go:1868:
01:19:43     c.Assert(result, icmd.Matches, icmd.Expected{ExitCode: 11})
01:19:43 ... result *cmd.Result = &cmd.Result{Cmd:(*exec.Cmd)(0xc0423846e0), ExitCode:0, Error:error(nil), Timeout:true, outBuffer:(*cmd.lockedBuffer)(0xc0439e9d40), errBuffer:(*cmd.lockedBuffer)(0xc0439e9dd0)} ("\nCommand: d:\\CI\\CI-cc2ee71\\binary\\docker.exe run -i --name test-inter-restart --restart=always busybox sh\nExitCode: 0 (timeout), Error: %!s(<nil>)\nStdout: \nStderr: \n")
01:19:43 ... expected cmd.Expected = cmd.Expected{ExitCode:11, Timeout:false, Error:"", Out:"", Err:""}
01:19:43 ... 
01:19:43 Command: d:\CI\CI-cc2ee71\binary\docker.exe run -i --name test-inter-restart --restart=always busybox sh
01:19:43 ExitCode: 0 (timeout), Error: %!s(<nil>)
01:19:43 Stdout: 
01:19:43 Stderr: 
01:19:43 
01:19:43 Failures:
01:19:43 ExitCode was 0 expected 11
01:19:43 Expected command to finish, but it hit the timeout

Merging: thanks @mlaventure, and thanks to all the reviewers ❤️

@icecrime icecrime merged commit 86de7c0 into moby:master Oct 1, 2016

4 of 5 checks passed

windowsRS1 Jenkins build Docker-PRs-WoW-RS1 3993 has failed
Details
docker/dco-signed All commits signed
Details
documentation success
Details
experimental Jenkins build Docker-PRs-experimental 24556 has succeeded
Details
janky Jenkins build Docker-PRs 33151 has succeeded
Details

@thaJeztah thaJeztah referenced this pull request Oct 1, 2016

Closed

docker rmi --unused #9054

@thaJeztah thaJeztah referenced this pull request Oct 20, 2016

Open

orphaned diffs #22207

@rosskevin rosskevin referenced this pull request in alienfast/docker-rails Oct 24, 2016

Open

Cleanup of images #21

@soredake soredake referenced this pull request in spotify/docker-gc Oct 30, 2016

Closed

docker 1.13 new features #136

bfirsh added a commit to bfirsh/docker that referenced this pull request Nov 9, 2016

Add prune endpoints to swagger.yaml
Introduced in #26108

Signed-off-by: Ben Firshman <ben@firshman.co.uk>

bfirsh added a commit to bfirsh/docker that referenced this pull request Nov 10, 2016

Add prune endpoints to swagger.yaml
Introduced in #26108

Signed-off-by: Ben Firshman <ben@firshman.co.uk>

@astropuffin astropuffin referenced this pull request Nov 16, 2016

Closed

Prune based on age #28497

@maxharlow maxharlow referenced this pull request in maxharlow/newsagent Nov 20, 2016

Closed

Cleanup dangling images when agent creation fails #1

@mmlkrx

This comment has been minimized.

Show comment
Hide comment
@mmlkrx

mmlkrx Dec 13, 2016

I'm having issues with docker system prune on Mac. It doesn't seem to actually make the pruned files available as storage in macOS Sierra. When I check About this Mac > Storage I just have a lot of "system" storage blocked. Any ideas how to free this up?

mmlkrx commented Dec 13, 2016

I'm having issues with docker system prune on Mac. It doesn't seem to actually make the pruned files available as storage in macOS Sierra. When I check About this Mac > Storage I just have a lot of "system" storage blocked. Any ideas how to free this up?

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Dec 13, 2016

Member

@mmlkrx the docker daemon on docker for Mac runs inside a VM with its own filesystem; running docker system prune will cleanup space inside that VM, but due to limitations of OS X, that won't shrink the size of the disk that the VM uses as it's filesystem, for more information about this, see docker/for-mac#371

Member

thaJeztah commented Dec 13, 2016

@mmlkrx the docker daemon on docker for Mac runs inside a VM with its own filesystem; running docker system prune will cleanup space inside that VM, but due to limitations of OS X, that won't shrink the size of the disk that the VM uses as it's filesystem, for more information about this, see docker/for-mac#371

@mmlkrx

This comment has been minimized.

Show comment
Hide comment
@mmlkrx

mmlkrx Dec 13, 2016

@thaJeztah Thanks for the quick reply, things make sense now :)!

mmlkrx commented Dec 13, 2016

@thaJeztah Thanks for the quick reply, things make sense now :)!

@michael-k

This comment has been minimized.

Show comment
Hide comment
@michael-k

michael-k Jan 20, 2017

Contributor

Why does it delete named volumes? Why not treat named volumes like named images?

Contributor

michael-k commented Jan 20, 2017

Why does it delete named volumes? Why not treat named volumes like named images?

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Jan 20, 2017

Member

@michael-k there is no real distinction between name and unnamed volumes; volumes don't have ID's only names; if you don't provide a name, a random name is generated (it looks like an ID, but is a name).

This PR is only the start though, and we can add further enhancements (a --filter option is already being worked on), so feel free to open a separate feature request (be sure to search first, there may be an existing one)

Member

thaJeztah commented Jan 20, 2017

@michael-k there is no real distinction between name and unnamed volumes; volumes don't have ID's only names; if you don't provide a name, a random name is generated (it looks like an ID, but is a name).

This PR is only the start though, and we can add further enhancements (a --filter option is already being worked on), so feel free to open a separate feature request (be sure to search first, there may be an existing one)

@michael-k

This comment has been minimized.

Show comment
Hide comment
@michael-k

michael-k Jan 20, 2017

Contributor

Thanks for the explanation 👍

Contributor

michael-k commented Jan 20, 2017

Thanks for the explanation 👍

@apetresc

This comment has been minimized.

Show comment
Hide comment
@apetresc

apetresc Jan 30, 2017

Just to clarify, which milestone is this scheduled for? I see it didn't make it into the 1.12.6 release, but is it on track for the 1.13.0 one?

EDIT: Ah, I see it is in the 1.13.0 changelog. Great to see :)

apetresc commented Jan 30, 2017

Just to clarify, which milestone is this scheduled for? I see it didn't make it into the 1.12.6 release, but is it on track for the 1.13.0 one?

EDIT: Ah, I see it is in the 1.13.0 changelog. Great to see :)

@mlaventure mlaventure deleted the mlaventure:data-mngt branch Jan 30, 2017

@qwIvan

This comment has been minimized.

Show comment
Hide comment

qwIvan commented Feb 3, 2017

good

@AlbanMontaigu AlbanMontaigu referenced this pull request in AlbanMontaigu/boot2docker-vagrant-extension Feb 23, 2017

Open

Use of docker system prune for gc ? #5

@dliappis dliappis referenced this pull request in elastic/elasticsearch-docker Mar 27, 2017

Closed

Child docker image failed on 4 or more layers #44

dnephin pushed a commit to dnephin/docker that referenced this pull request Apr 17, 2017

Arnaud Porterie
Merge pull request #26108 from mlaventure/data-mngt
New Data Management commands
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment