Failed Loading Environment error after rebooting manager node

[joshbuker]

Bug description
After rebooting the manager node in a 1 manager, 1 worker docker swarm setup, portainer will boot the service, but attempting to load the primary environment will fail with an error:

Failed loading environment
Unable to find an agent on any manager node

This error happens both from the worker and manager node instances of the portainer web interface.

Expected behavior
After a reboot, portainer boots normally and allows loading the environment.

Portainer Logs
Provide the logs of your Portainer container or Service.

Steps to reproduce the issue:

1. Setup a docker swarm with 1 manager, 1 worker. The manager is Ubuntu Server in my case, while the worker is Pop OS!
2. Verify that portainer is working normally by visiting the web interface
3. Reboot the manager node (system reboot)
4. Visit web interface again, attempt clicking on primary environment
5. See error

Additional context
Discord thread has additional context if needed.

[tamarahenson]

@joshbuker

Thank you for logging this request that was initially posted in our Discord channel. I am going to provide further updates here with my testing and forward to Product for review.

Thanks!

[joshbuker]

Per our Discord conversation, using sudo systemctl restart docker on the worker node is confirmed as a workaround for the issue. After rebooting the service, the error goes away and the GUI can be accessed again - without any changes on the manager node.

[tamarahenson]

DOCKER SWARM REBOOT TESTING:

(1) Manager, (1+) Worker

[1] Install Swarm Portainer Server on Manager:
curl -L https://downloads.portainer.io/ee2-16/portainer-agent-stack.yml -o portainer-agent-stack.yml
docker stack deploy -c portainer-agent-stack.yml portainer

[2] Reboot Manager Node
[3] Navigate UI
[4] Wait 2 Minutes
https://user-images.githubusercontent.com/20426210/210469483-0e6c40c9-3325-4601-8fcf-3c11100d9b33.mov
[5] Toast appears:

Failed loading environment
Unable to find an agent on any manager node

[6] Home >> Environment:primary >> Node indicator is no longer appearing

WORKAROUND:
[1] sudo systemctl restart docker on worker node
[2] Navigate UI on Manager
[3] Wait 1 minute.
[4] Home >> Environment:primary >> Node indicator is appearing with node counts

[5] Toast no longer appears

Note: this workaround also works on multiple worker node clusters. You only need to restart docker on one worker node.

ADDITIONAL TESTING:

(2) Manager, (1) Worker
Notes: Issue does not occur.

TODO:
Need to determine if this is Docker Swarm behavior or a bug. Updates to follow.

Thanks!

[tamarahenson]

Update:

Upon further investigation, I have found a better workaround.

Run the following on the Manager node (image name may need to be edited):

sudo docker service update --image portainer/agent:2.16.2 --force portainer_agent

Thanks!

[vampywiz17]

same problem, the workaround working... but a really annoying that this issue are exist more than one year...

[CeamKrier]

Per our Discord conversation, using sudo systemctl restart docker on the worker node is confirmed as a workaround for the issue. After rebooting the service, the error goes away and the GUI can be accessed again - without any changes on the manager node.

This resolved my issue

[bdoublet91]

Hi,
I'm using portainer for 4 years in production swarm env. I have always seen this issue and the only workaround is to restart portainer_agent on all nodes ... quite annoying. And come again few days later
I think portainer doesnt support well network latency / reconnections and also I can't use portainer in a multicloud swarm env with 5ms latency because the agent node timeout half the time...
I use the latest version 2.19.4

So now I restart portainer in a cronjob at 23:00pm every day to limit this issue ...
Please, give us a fix for that

thanks

[BloodBlight]

This has been an issue for years now. Has any progress been made? :/

We have written a cron job that looks fore the "unable to redirect request to a manager node: no manager node found" then kicks off a service update, but it is not really a great solution...

[whysi]

we are on the same boat, this happen in every environment that we have (Portainer BE customer) after we reboot any node in a swarm cluster, not necessairly a manager node. This is quite frustrating and we have it since the beginning of our experience with Portainer.
We tried to configure a systemd unit that restart the portainer_agent service on one manager for each swarm cluster 40 minutes after the node booted up (that's pretty much the time it takes to update one of our clusters plus 20 minutes of margin) but is not a real solution. For example if we have to restart any worker node for an unplanned maintenance then we have to remember also to manually restart the portainer_agent service for that swarm cluster.

@tamarahenson Any plan to fix this? The workaround is not a real solution because we have different reboot/update policies in our swarm clusters. If that's not on roadmap we will open a support case on the customer portal because we are having many complaints from out end users that are perceiving Portainer as unstable and broken every time they have this errors. I know that the workaround is a quick fix but at that point the user has already experienced the error.