New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dokku cleanup removed all "exited" containers #1220

Closed
palamccc opened this Issue Jun 7, 2015 · 31 comments

Comments

Projects
None yet
8 participants
@palamccc

palamccc commented Jun 7, 2015

dokku should only remove containers created by dokku. It should not remove containers created by others.

I'm running host, with dokku and other docker images. I use 'data-only' container pattern to store data in a 'non-running' container. But dokku removes my non-running data container as well.
http://container42.com/2013/12/16/persistent-volumes-with-docker-container-as-volume-pattern/

https://github.com/progrium/dokku/blob/4aedb6507a3bdec1ff29be2ff60366075186a72e/dokku#L170

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Jun 7, 2015

Member

How are we to detect that you're running a non-dokku provided container?

Member

josegonzalez commented Jun 7, 2015

How are we to detect that you're running a non-dokku provided container?

@3onyc

This comment has been minimized.

Show comment
Hide comment
@3onyc

3onyc Jun 8, 2015

Contributor

Possible solution is to label (--label) containers created with Dokku, then modify clean-up to only remove those. Note that any plug-ins need to be modified to also label their containers.

You'd need to modify dokku itself to accomplish this though.

Contributor

3onyc commented Jun 8, 2015

Possible solution is to label (--label) containers created with Dokku, then modify clean-up to only remove those. Note that any plug-ins need to be modified to also label their containers.

You'd need to modify dokku itself to accomplish this though.

@palamccc

This comment has been minimized.

Show comment
Hide comment
@palamccc

palamccc Jun 9, 2015

Is it possible to run all docker build commands with "--rm" option, it will automatically remove container after doing the work. so there won't be any need for cleanup.

palamccc commented Jun 9, 2015

Is it possible to run all docker build commands with "--rm" option, it will automatically remove container after doing the work. so there won't be any need for cleanup.

@3onyc

This comment has been minimized.

Show comment
Hide comment
@3onyc

3onyc Jun 9, 2015

Contributor

That would potentially leave images and containers, for example the Docker daemon crashing would prevent containers (with --rm) from being deleted.

Contributor

3onyc commented Jun 9, 2015

That would potentially leave images and containers, for example the Docker daemon crashing would prevent containers (with --rm) from being deleted.

@michaelshobbs

This comment has been minimized.

Show comment
Hide comment
@michaelshobbs

michaelshobbs Jun 12, 2015

Member

it's also very useful to have the intermediate images/containers around debug deployments. Dokku was never (from what I can tell) specifically written to "play nice" with non-dokku managed images and containers.

Having said that, we can definitely label dokku core containers but as @3onyc mentioned, plugins will also need to be modified.

Anybody else have thoughts on this?

Member

michaelshobbs commented Jun 12, 2015

it's also very useful to have the intermediate images/containers around debug deployments. Dokku was never (from what I can tell) specifically written to "play nice" with non-dokku managed images and containers.

Having said that, we can definitely label dokku core containers but as @3onyc mentioned, plugins will also need to be modified.

Anybody else have thoughts on this?

@zimbatm

This comment has been minimized.

Show comment
Hide comment
@zimbatm

zimbatm Jun 18, 2015

I have just been hit by that issue. Maybe the doc should say more clearly that dokku is not meant to work with other docker containers.

zimbatm commented Jun 18, 2015

I have just been hit by that issue. Maybe the doc should say more clearly that dokku is not meant to work with other docker containers.

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Jun 24, 2015

Member

@michaelshobbs Not sure I understand your first comment:

it's also very useful to have the intermediate images/containers around debug deployments. Dokku was never (from what I can tell) specifically written to "play nice" with non-dokku managed images and containers.

In any case, I would rather we just state in the docs that dokku doesn't play nice with externally-managed containers. We should definitely not delete data-only containers, though I'm not sure what we can do here. Maybe support a special, dokku-specific label that we'll use to ensure we don't kill certain containers?

Member

josegonzalez commented Jun 24, 2015

@michaelshobbs Not sure I understand your first comment:

it's also very useful to have the intermediate images/containers around debug deployments. Dokku was never (from what I can tell) specifically written to "play nice" with non-dokku managed images and containers.

In any case, I would rather we just state in the docs that dokku doesn't play nice with externally-managed containers. We should definitely not delete data-only containers, though I'm not sure what we can do here. Maybe support a special, dokku-specific label that we'll use to ensure we don't kill certain containers?

@alessio

This comment has been minimized.

Show comment
Hide comment
@alessio

alessio Jun 30, 2015

Contributor

👍 for containers labelling

Contributor

alessio commented Jun 30, 2015

👍 for containers labelling

josegonzalez added a commit that referenced this issue Sep 6, 2015

Do not remove any containers that have the label `dokku`. Closes #1220
This can be used to mark data containers as non-removable by the cleanup function
@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Sep 6, 2015

Member

I recently needed to implement this feature in order to not kill dokku service containers on startup (or whenever dokku cleanup is run). Checkout #1444 for more details.

/cc @Flink

Member

josegonzalez commented Sep 6, 2015

I recently needed to implement this feature in order to not kill dokku service containers on startup (or whenever dokku cleanup is run). Checkout #1444 for more details.

/cc @Flink

@Cactusbone

This comment has been minimized.

Show comment
Hide comment
@Cactusbone

Cactusbone Sep 7, 2015

Contributor

not fan of the "dokku" label, imo it means it's managed by dokku, when it's not. I don't have another suggestion though :/ anyway this is just what i needed :) any date for next release ? :)

Contributor

Cactusbone commented Sep 7, 2015

not fan of the "dokku" label, imo it means it's managed by dokku, when it's not. I don't have another suggestion though :/ anyway this is just what i needed :) any date for next release ? :)

@jazzzz

This comment has been minimized.

Show comment
Hide comment
@jazzzz

jazzzz Sep 7, 2015

Contributor

it's also very useful to have the intermediate images/containers around debug deployments. Dokku was never (from what I can tell) specifically written to "play nice" with non-dokku managed images and containers.

Having said that, we can definitely label dokku core containers but as @3onyc mentioned, plugins will also need to be modified.

OTOH, now I have to change all the tools I use beside dokku to add a dokku label to every images and container.

I think it would be better if dokku could play nice with non-dokku images and containers.

Contributor

jazzzz commented Sep 7, 2015

it's also very useful to have the intermediate images/containers around debug deployments. Dokku was never (from what I can tell) specifically written to "play nice" with non-dokku managed images and containers.

Having said that, we can definitely label dokku core containers but as @3onyc mentioned, plugins will also need to be modified.

OTOH, now I have to change all the tools I use beside dokku to add a dokku label to every images and container.

I think it would be better if dokku could play nice with non-dokku images and containers.

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Sep 7, 2015

Member

@Cactusbone No, I take that label to mean that it is related to dokku. Personally I assume anything in docker is somehow managed by dokku, and if it's not, then that's up to the user of dokku to figure out. This is just giving you all a simple way to skip those containers.

@jazzzz We could disable the cleanup command and let you self-manage that, but then we'll use more disk space/memory and users will have other issues. Do you have an alternative solution for this problem? Also, what other tools are you using in conjunction with dokku on the same server?

Member

josegonzalez commented Sep 7, 2015

@Cactusbone No, I take that label to mean that it is related to dokku. Personally I assume anything in docker is somehow managed by dokku, and if it's not, then that's up to the user of dokku to figure out. This is just giving you all a simple way to skip those containers.

@jazzzz We could disable the cleanup command and let you self-manage that, but then we'll use more disk space/memory and users will have other issues. Do you have an alternative solution for this problem? Also, what other tools are you using in conjunction with dokku on the same server?

@jazzzz

This comment has been minimized.

Show comment
Hide comment
@jazzzz

jazzzz Sep 8, 2015

Contributor

@josegonzalez I use many images and containers beside dokku: some are created and started with shells scripts, some with ansible, some directly from Docker Hub, some are manually created... It would make dokku much less useful for me if dokku needs to manage everything in docker.

I think it would be better to tag dokku related containers and images with the dokku instead of the reverse: I would be misleading to see the containers and images not related to dokku with the dokkulabel. And I don't think many plugins would be broken (if any).

Anyway, a plugin with a data-only container will need to set the dokku label on this container but not on the others: it feels strange.

Contributor

jazzzz commented Sep 8, 2015

@josegonzalez I use many images and containers beside dokku: some are created and started with shells scripts, some with ansible, some directly from Docker Hub, some are manually created... It would make dokku much less useful for me if dokku needs to manage everything in docker.

I think it would be better to tag dokku related containers and images with the dokku instead of the reverse: I would be misleading to see the containers and images not related to dokku with the dokkulabel. And I don't think many plugins would be broken (if any).

Anyway, a plugin with a data-only container will need to set the dokku label on this container but not on the others: it feels strange.

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Sep 8, 2015

Member

@jazzzz I still don't understand how you would want dokku cleanup to work. We can either cleanup containers or we don't. In either case, someone is going to have issues.

Member

josegonzalez commented Sep 8, 2015

@jazzzz I still don't understand how you would want dokku cleanup to work. We can either cleanup containers or we don't. In either case, someone is going to have issues.

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Sep 8, 2015

Member

As well, we are only using this to remove exited containers. So if your container is running,it sticks around.

Member

josegonzalez commented Sep 8, 2015

As well, we are only using this to remove exited containers. So if your container is running,it sticks around.

@jazzzz

This comment has been minimized.

Show comment
Hide comment
@jazzzz

jazzzz Sep 8, 2015

Contributor

@josegonzalez I would prefer if dokku only remove exited containers with the dokku label, I think it would be safer and cleaner.

With the current implementation, if I use a docker container with a database in a data-only container (with a dokku plugin or not), because this container is not running, dokku will delete this container and I will lose my data.

Contributor

jazzzz commented Sep 8, 2015

@josegonzalez I would prefer if dokku only remove exited containers with the dokku label, I think it would be safer and cleaner.

With the current implementation, if I use a docker container with a database in a data-only container (with a dokku plugin or not), because this container is not running, dokku will delete this container and I will lose my data.

@michaelshobbs

This comment has been minimized.

Show comment
Hide comment
@michaelshobbs

michaelshobbs Sep 8, 2015

Member

(Preface: IMHO)
I will re-iterate here that dokku is a docker container manager and there is no way to cleanly have dokku manage its containers and not screw something up for some edge case somewhere. The project has taken a conscious direction of optimizing for the 'simplest' deployment (see DO droplet for example) where dokku manages all docker containers on a given host. Sharing the docker daemon is considered an 'advanced' deployment and is to be handled by the managing user.

@josegonzalez what exactly needed to re-implemented for service containers? if a service container is running it doesn't get stopped or removed and i'd hope that service containers are using docker volumes for persistence in case the container does get 'accidentally' cleaned up by some other (user's or otherwise) action.

Member

michaelshobbs commented Sep 8, 2015

(Preface: IMHO)
I will re-iterate here that dokku is a docker container manager and there is no way to cleanly have dokku manage its containers and not screw something up for some edge case somewhere. The project has taken a conscious direction of optimizing for the 'simplest' deployment (see DO droplet for example) where dokku manages all docker containers on a given host. Sharing the docker daemon is considered an 'advanced' deployment and is to be handled by the managing user.

@josegonzalez what exactly needed to re-implemented for service containers? if a service container is running it doesn't get stopped or removed and i'd hope that service containers are using docker volumes for persistence in case the container does get 'accidentally' cleaned up by some other (user's or otherwise) action.

@jazzzz

This comment has been minimized.

Show comment
Hide comment
@jazzzz

jazzzz Sep 8, 2015

Contributor

Not sure if I was clear, I was talking about data volume containers. These containers are not running, thus dokku removes them.

Contributor

jazzzz commented Sep 8, 2015

Not sure if I was clear, I was talking about data volume containers. These containers are not running, thus dokku removes them.

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Sep 8, 2015

Member

@michaelshobbs they persist data and config onto disk (in /var/lib/dokku/services/mysql/service-name for instance). Is that what you mean? I personally like the idea of the same container being started and not being tossed just because the machine rebooted and we are running dokku cleanup.

Re: container-management, I agree that I see dokku as the sole-arbiter of containers on a given host. The simplest thing I can think of is allowing people the ability to disable dokku cleanup and let them handle cleanup on their own.

@jazzzz those containers are running for me?

root@dokku:~# docker create -v /dbdata --name dbdata gliderlabs/herokuish /bin/true
7c4916b095369ba27779fe1d1842215255d563c0f2805fd0125762c11f3ffb98
root@dokku:~# docker ps -a
CONTAINER ID        IMAGE                         COMMAND             CREATED             STATUS              PORTS               NAMES
7c4916b09536        gliderlabs/herokuish:latest   "/bin/true"         2 seconds ago                                               dbdata
Member

josegonzalez commented Sep 8, 2015

@michaelshobbs they persist data and config onto disk (in /var/lib/dokku/services/mysql/service-name for instance). Is that what you mean? I personally like the idea of the same container being started and not being tossed just because the machine rebooted and we are running dokku cleanup.

Re: container-management, I agree that I see dokku as the sole-arbiter of containers on a given host. The simplest thing I can think of is allowing people the ability to disable dokku cleanup and let them handle cleanup on their own.

@jazzzz those containers are running for me?

root@dokku:~# docker create -v /dbdata --name dbdata gliderlabs/herokuish /bin/true
7c4916b095369ba27779fe1d1842215255d563c0f2805fd0125762c11f3ffb98
root@dokku:~# docker ps -a
CONTAINER ID        IMAGE                         COMMAND             CREATED             STATUS              PORTS               NAMES
7c4916b09536        gliderlabs/herokuish:latest   "/bin/true"         2 seconds ago                                               dbdata
@jazzzz

This comment has been minimized.

Show comment
Hide comment
@jazzzz

jazzzz Sep 8, 2015

Contributor

@josegonzalez for me it's not running, Try with docker ps (without -a). Also STATUS is empty.

Contributor

jazzzz commented Sep 8, 2015

@josegonzalez for me it's not running, Try with docker ps (without -a). Also STATUS is empty.

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Sep 8, 2015

Member

Interesting, I guess if there is no status, that is treated as exited by docker. What fun!

Member

josegonzalez commented Sep 8, 2015

Interesting, I guess if there is no status, that is treated as exited by docker. What fun!

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Sep 8, 2015

Member

Seems like there isn't a way to check if a container is a data-only container, as they are a convention, not an actual docker construct.

Member

josegonzalez commented Sep 8, 2015

Seems like there isn't a way to check if a container is a data-only container, as they are a convention, not an actual docker construct.

@jazzzz

This comment has been minimized.

Show comment
Hide comment
@jazzzz

jazzzz Sep 8, 2015

Contributor

Seems like there isn't a way to check if a container is a data-only container.

Yeah, that's why I suggest that dokku creates a dokku label for the containers it creates and only the exited containers with this label. Are there really plugins that depend on dokku cleanup?

Contributor

jazzzz commented Sep 8, 2015

Seems like there isn't a way to check if a container is a data-only container.

Yeah, that's why I suggest that dokku creates a dokku label for the containers it creates and only the exited containers with this label. Are there really plugins that depend on dokku cleanup?

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Sep 8, 2015

Member

We don't want to remove the service plugins on boot of the server, hence the current hack.

Member

josegonzalez commented Sep 8, 2015

We don't want to remove the service plugins on boot of the server, hence the current hack.

josegonzalez added a commit that referenced this issue Sep 8, 2015

Revert change that whitelisted dokku-labeled containers
Rather than having a one-off hack for "blessed" containers, we should solve the issue properly. As that would take a bit more work, reverting to the previous state is preferred.

Better that we solve a problem problem than with a hack that we'll need to roll back in the future.

Refs #1220

[ci skip]

josegonzalez added a commit that referenced this issue Sep 8, 2015

Revert change that whitelisted dokku-labeled containers
Rather than having a one-off hack for "blessed" containers, we should solve the issue properly. As that would take a bit more work, reverting to the previous state is preferred.

Better that we solve a problem problem than with a hack that we'll need to roll back in the future.

Refs #1220

[ci skip]

josegonzalez added a commit that referenced this issue Sep 9, 2015

Revert change that whitelisted dokku-labeled containers
Rather than having a one-off hack for "blessed" containers, we should solve the issue properly. As that would take a bit more work, reverting to the previous state is preferred.

Better that we solve a problem problem than with a hack that we'll need to roll back in the future.

Refs #1220

[ci skip]

josegonzalez added a commit that referenced this issue Sep 10, 2015

Revert change that whitelisted dokku-labeled containers
Rather than having a one-off hack for "blessed" containers, we should solve the issue properly. As that would take a bit more work, reverting to the previous state is preferred.

Better that we solve a problem problem than with a hack that we'll need to roll back in the future.

Refs #1220

[ci skip]

josegonzalez added a commit that referenced this issue Sep 15, 2015

Revert change that whitelisted dokku-labeled containers
Rather than having a one-off hack for "blessed" containers, we should solve the issue properly. As that would take a bit more work, reverting to the previous state is preferred.

Better that we solve a problem problem than with a hack that we'll need to roll back in the future.

Refs #1220

[ci skip]

josegonzalez added a commit that referenced this issue Sep 15, 2015

Revert change that whitelisted dokku-labeled containers
Rather than having a one-off hack for "blessed" containers, we should solve the issue properly. As that would take a bit more work, reverting to the previous state is preferred.

Better that we solve a problem problem than with a hack that we'll need to roll back in the future.

Refs #1220

[ci skip]

@josegonzalez josegonzalez reopened this Oct 16, 2015

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Jan 3, 2016

Member

@michaelshobbs we can probably add the dokku label via the docker-options plugin. At a later date - probably 0.5.0? - we can update dokku cleanup to only take those with said label into account when running cleanup?

Member

josegonzalez commented Jan 3, 2016

@michaelshobbs we can probably add the dokku label via the docker-options plugin. At a later date - probably 0.5.0? - we can update dokku cleanup to only take those with said label into account when running cleanup?

@michaelshobbs

This comment has been minimized.

Show comment
Hide comment
@michaelshobbs

michaelshobbs Jan 5, 2016

Member

Yeah works for me

Member

michaelshobbs commented Jan 5, 2016

Yeah works for me

@michaelshobbs

This comment has been minimized.

Show comment
Hide comment
@michaelshobbs

michaelshobbs Jan 5, 2016

Member

@josegonzalez plugins would need to follow this as well, right?

Member

michaelshobbs commented Jan 5, 2016

@josegonzalez plugins would need to follow this as well, right?

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Jan 5, 2016

Member

We're only going to cleanup apps started/stopped by dokku. If a plugin wants to opt into that cleanup, up to them.

For example, the official service plugins shouldn't (because we might accidentally cleanup a stopped container during a deploy).

Member

josegonzalez commented Jan 5, 2016

We're only going to cleanup apps started/stopped by dokku. If a plugin wants to opt into that cleanup, up to them.

For example, the official service plugins shouldn't (because we might accidentally cleanup a stopped container during a deploy).

@michaelshobbs

This comment has been minimized.

Show comment
Hide comment
@michaelshobbs

michaelshobbs Jan 5, 2016

Member

Ok I noted in the docs of #1828 how to do it and why you might want. The language might need to be cleaned up a bit.

Member

michaelshobbs commented Jan 5, 2016

Ok I noted in the docs of #1828 how to do it and why you might want. The language might need to be cleaned up a bit.

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Mar 9, 2016

Member

@michaelshobbs worth adding this to 0.5.x or waiting till 0.6.x?

Member

josegonzalez commented Mar 9, 2016

@michaelshobbs worth adding this to 0.5.x or waiting till 0.6.x?

@michaelshobbs

This comment has been minimized.

Show comment
Hide comment
@michaelshobbs

michaelshobbs Mar 9, 2016

Member

@josegonzalez your call. PR created 😄

Member

michaelshobbs commented Mar 9, 2016

@josegonzalez your call. PR created 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment