Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove containers on swarm service scale down #1372

Closed
igrcic opened this issue Aug 15, 2016 · 16 comments

Comments

Projects
None yet
5 participants
@igrcic
Copy link

commented Aug 15, 2016

Hi everyone,

first I'd like to ask if this is the right repo to be asking questions regarding to docker swarm * engine commands?

I am currently testing docker swarm engine in order to see if I can use this new functionality for some of our (non-critical) production environment.

I noticed that when scaling down services, containers in exited status remain (1.12) in docker. When I was testing beta2 or beta3 the containers were automatically removed.

Can one achieve this behavior somehow in v1.12?

Thank you,
Ivan

@igrcic igrcic changed the title Remove containers on scale down Remove containers on swarm service scale down Aug 15, 2016

@aaronlehmann

This comment has been minimized.

Copy link
Collaborator

commented Aug 15, 2016

By default, we leave the most recent 5 containers per replica in place to aid troubleshooting (for example, providing the ability to execute a shell in one of these containers and inspect logs). This is a configurable setting, so you could set the number of containers to retain to 0 with docker swarm update --task-history-limit 0. This should provide the behavior you're looking for. Note that it will cause exited or failed tasks not to show up in docker service ps, since those entries are removed when the containers are cleaned up.

cc @sfsmithcha - not sure if this is well-documented.

@stevvooe

This comment has been minimized.

Copy link
Contributor

commented Aug 16, 2016

@igrcic Note that it is also safe to remove exited containers manually. Something like the following can run on the work nodes periodically:

$ docker ps -qa -f label=com.docker.swarm.task -f status=exited | xargs docker rm -f

This can be used if you'd like to have a more aggressive policy about cleaning up resources but would like to have some task history for debugging purposes.

@igrcic

This comment has been minimized.

Copy link
Author

commented Aug 17, 2016

Thank you @aaronlehmann,

i have tried with --task-history-limit 0, but it doesnt really do anything. Scaling from 7 to 1 instances leaves 6 exited ones.

image
Docker version 1.12.0, build 8eab29e

@stevvooe tnx i'm already doing that. I just thought that TaskReaper thingy can do it for us though ;)

@aaronlehmann

This comment has been minimized.

Copy link
Collaborator

commented Aug 17, 2016

I know there is a bug where --task-history-limit 0 is not honored as an argument to docker swarm init. I'd expect it to work with docker swarm update, though. If it isn't, we should investigate that. Did you update the swarm or create a new one?

@igrcic

This comment has been minimized.

Copy link
Author

commented Aug 17, 2016

Hi,

i just tried with both, update and init. Task history limit is not honored (nor is the default value of 5)

@igrcic

This comment has been minimized.

Copy link
Author

commented Aug 17, 2016

I guess its related to moby/moby#24394

@aaronlehmann

This comment has been minimized.

Copy link
Collaborator

commented Aug 17, 2016

The limit is per replica, so you wouldn't see the default limit of 5 come into play until at least one replica restarts 5 times.

When you try with a limit of 0, do the old scaled-down tasks disappear from docker service ps?

@stevvooe

This comment has been minimized.

Copy link
Contributor

commented Aug 17, 2016

@igrcic For the most part, orphan containers have been mitigated. There may still be a slight race condition described in moby/moby#24858, but it should be unrealistic.

In addition to troubleshooting suggested by @aaronlehmann, it would also be good to get the output of the logs during the period when you expect removal to happen. There could be some condition on your hosts that are preventing the removal from proceeding and we'd need to track that down.

@igrcic

This comment has been minimized.

Copy link
Author

commented Aug 18, 2016

tnx @aaronlehmann now i see what you mean, this actually applies for service replica history count (docker service ps {serviceName})

In that case, no it doesnt work for me, it always stays at 4+1 replicas
image

@stevvooe the only thing I can see in logs is:

time="2016-08-18T17:37:45.738067694+02:00" level=info msg="Failed to delete real server 10.255.0.39 for vip 10.255.0.8 fwmark 260: no such file or directory" 
time="2016-08-18T17:37:45.738237636+02:00" level=info msg="Failed to delete real server 10.255.0.39 for vip 10.255.0.8 fwmark 260: no such file or directory" 
time="2016-08-18T17:37:45.738683745+02:00" level=info msg="Failed to delete real server 10.255.0.19 for vip 10.255.0.8 fwmark 260: no such file or directory" 
time="2016-08-18T17:37:45.738881686+02:00" level=info msg="Failed to delete real server 10.255.0.19 for vip 10.255.0.8 fwmark 260: no such file or directory" 
time="2016-08-18T17:37:45.739098041+02:00" level=info msg="Failed to delete real server 10.255.0.34 for vip 10.255.0.8 fwmark 260: no such file or directory" 
time="2016-08-18T17:37:45.739244895+02:00" level=info msg="Failed to delete real server 10.255.0.34 for vip 10.255.0.8 fwmark 260: no such file or directory" 
time="2016-08-18T17:37:45.739463884+02:00" level=info msg="Failed to delete real server 10.255.0.28 for vip 10.255.0.8 fwmark 260: no such file or directory" 
time="2016-08-18T17:37:45.739611479+02:00" level=info msg="Failed to delete real server 10.255.0.28 for vip 10.255.0.8 fwmark 260: no such file or directory" 

Do not know if that is related though.

Tnx,
Ivan

@aaronlehmann

This comment has been minimized.

Copy link
Collaborator

commented Aug 18, 2016

I tried this out and changing --task-history-limit seems to work for me. Note that the change won't take effect instantly - tasks need to restart for old containers to get cleaned up according to the new limit. But when I create a swarm with a task history limit set, it looks like that limit is respected. And when I update that limit with swarm update, that's also respected as soon as the a task restarts.

BTW, I misremembered how the limit is applied. It actually counts all tasks for a given instance, not just the old tasks. So --task-history-limit 1 should do what you want (although 0 should do the same thing).

@aluzzardi

This comment has been minimized.

Copy link
Contributor

commented Aug 28, 2016

@aaronlehmann Should we actually nuke tasks when we scale down? No point in keeping the history for those I guess?

Slot history makes sense for crashes, rolling updates, etc but when you scale down they're pretty useless.

@aaronlehmann

This comment has been minimized.

Copy link
Collaborator

commented Aug 29, 2016

The only reason I can think of to keep them is to show a record of the scaling itself. It's not a very strong reason. Deleting the tasks immediately when scaling down would probably be fine.

@stevvooe

This comment has been minimized.

Copy link
Contributor

commented Aug 29, 2016

@aaronlehmann @aluzzardi Delete or not, these can be removed from the assignment set.

@igrcic

This comment has been minimized.

Copy link
Author

commented Sep 26, 2016

Hi all,

+1 for nuking the tasks :)

@stevvooe can you explain what do you mean by that?

@stevvooe

This comment has been minimized.

Copy link
Contributor

commented Sep 26, 2016

@igrcic We dispatch tasks by maintaining an "Assignment Set" to the target node. The dispatcher protocol maintains this assignment set between the worker and manager. If a task is outside the assignment set, the node can choose to remove the resources associated with that task (ie delete the container).

The point here is, the discussion of deletion is moot. All that needs to be done from the perspective of the work is to not include these tasks in the assignment set and the node will delete them. The manager can choose to keep them around or delete them separately.

@aluzzardi

This comment has been minimized.

Copy link
Contributor

commented Sep 26, 2016

The consensus is to reduce the number of slots upon scale down

/cc @aaronlehmann

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.