Automatically hide non-removable "ghost" tasks

[mrmachine]

Is your feature request related to a problem? Please describe.

Docker swarm mode has a bug where vestigial "ghost" tasks on nodes that have been removed from the swarm are never removed from the task history.

• These show up in portainer as "running" or "starting" or whatever their state was at the time the node was removed, forever.
• They show up as not belonging to any node.
• docker swarm update --task-history-limit 0 (or 1 or 5) does not fix the problem.
• If the task history limit is equal to or less than the number of ghost tasks, you never see new legitimate tasks in the history.
• Portainer will show a scale like 3 / 1 running for affected services.

This is causing us a serious problem in being able to ascertain the running status of all our stacks, as we need carefully examine the detail of every task to find out if it is actually running or if only ghost tasks are still being reported as running.

Describe the solution you'd like

1. Portainer should not display tasks in task history lists when the associated node is not known to the swarm.
2. Portainer should ignore ghost tasks when it shows how many replicas are running, next to the scale link.
3. Portainer should add a "recreate stack" button that deletes and then immediately recreates an identical stack, to clear out these ghost tasks with minimal downtime. This might also be a remedy for other Docker swarm bugs.

Describe alternatives you've considered

I think the only way to remove these ghost tasks is to delete and recreate from scratch all stacks that contain affected services. For us this is around 20 production stacks that will experience downtime as a result.

Without a "recreate stack" button, I will need to manually copy and paste the stack name, stack file, and each environment variable name and value individually back into the form (while the site is down).

Some of our stacks have 15 environment variables. So that's over 30 copy and paste operations to recreate the stack and clear out these vestigial ghost tasks, while the site is down.

Additional context

I realise that the root problem is a bug in Docker swarm mode, not Portainer, but the impact of this bug seriously limits the usefulness of Portainer in being able to confidently (or at all) ascertain the running state of our stacks.

This issue has been open for some time with Docker already with not much traction. And even if it were fixed tomorrow, I fear the fix would be at least 3 months before it lands in Docker for AWS. Or more likely 6 months or even never, given that Docker for AWS is already a full stable release behind and Docker has switched to a twice-a-year release cadence.

Here's the open issue: moby/moby#34122

Here's a screenshot from Portainer:

Even if solutions 1 and 2 from above are rejected as workarounds for someone else's bug, 3 should at least allow us to get back to a normal state with as little downtime as possible and might also be useful in other scenarios.

[mrmachine]

Here's another screenshot, after scaling the above acme_nginx service down to 0, Portainer tells me there are 2 replicas running and with a quick look at this task list I would assume all is well.

Only when closely looking for a missing value in the node field plus more replicas than desired would I figure out that something is wrong.

[mrmachine]

Another another one, that would make me stop and think why does this service still have pending tasks?

[mrmachine]

And why did the previous task fail for this service (even though it actually completed successfully):

[sorenhansendk]

We have the same problems with our removed nodes.
We often add and remove nodes, so this is a very big issue for us.

[mrmachine]

@sorenhansendk can you add your experience to the issue covering the root cause in Docker swarm mode? moby/moby#34122

[sorenhansendk]

We have the same issue, with ghost tasks on nodes there is removed. The tasks still "running" but the nodes dosen't exist anymore - but Portainer still shows the tasks on the overview.

[linuxart]

I have same issue.

[xAt0mZ]

Counters will be fixed with PR #2127 , tasks list can be fixed with the same "hack" way (e.g. hiding tasks that have desired state != current state).
Ghosts tasks will still be there, but hidden in the UI.

[mrmachine]

FYI, I've discovered a way to remove the ghost tasks. Scale a service down to 0 then click the checkbox and then click Update button from the stack page. Updating a service recreates all of its tasks. When you do this and scale is >=1 the ghost tasks remain, but with scale 0 even though it's basically a no-op, it removes the ghost tasks.

[Diftraku]

After removing one node from a swarm a bit too hastily, I can vouch that hiding stuck/ghost tasks may be ill-advised, especially if you rely on using service names for routing connections (like HTTP from nginx service with a published port to a container belonging to a service with unpublished port, sharing the same network). Any "ghosted" tasks will still be considered as valid targets for the swarm's internal load balancing, regardless of their actual state (this took a while to figure out).

The workaround of scaling to 0, force updating and then scaling back does work, but it might require toggling back and forth with especially persistent ghost tasks.

[mrmachine]

It just took me about half an hour to review a list of 174 services looking for any where replicas > scale AND no node was defined for some of the replicas, scale them all down to zero (page reload every time), click update (page reload every time), scale back up (another page reload every time), and lots of scrolling a long list in between every click/reload...

if Portainer didn't reload after every scale/update, that would make this process WAY quicker and easier... Even better would be if Portainer had a button that did all this for me... Or perhaps this could be done automatically with a Docker image deployed to a manager node and bind mounting the docker socket... I doubt Docker will ever fix this bug in swarm...