Reboot and network race conditions - "Status: Exited Exited (255)"

[MrTengil]

To Reproduce

1. Spin up Dokploy on some server.
2. Deploy docker compose containing bunch of services.
3. Add atleast one of the services to external dokploy-network
4. Deploy it and make sure it's up and running
5. Force shutdown of server (not gracefully)
6. Boot up the server.
7. Check for any none starting applications: docker ps -a -f "status=exited"

Current vs. Expected behavior

Expectations: All of my deployed containers to spin up again with no issue after the server forcefully shuts down and boots up.
Reality: Some of my containers (the ones connected to dokploy-network) sometimes exits with status code 255
Even Dokploys own services fails and exits on 255 but new are spun up.

Exited:

• Exited (255) 23 minutes ago 6379/tcp dokploy-redis.1.<id_redis>
• Exited (255) 23 minutes ago 0.0.0.0:3000->3000/tcp, [::]:3000->3000/tcp dokploy.1.<id_dokploy>
• Exited (255) 23 minutes ago 0.0.0.0:3000->3000/tcp, [::]:3000->3000/tcp dokploy-postgres.1.<id_postgres>

Up:

• Up 23 minutes 6379/tcp dokploy-redis.1.<id_redis_up>
• Up 23 minutes 0.0.0.0:3000->3000/tcp, [::]:3000->3000/tcp dokploy.1.<id_dokploy_up>
• Up 23 minutes 5432/tcp dokploy-postgres.1.<id_postgres_up>

If I SSH into the server and restart the exited containers with docker restart $(docker ps -q -a -f "status=exited") they spin up just fine.

Looking at the logs, sudo journalctl -u docker -n 100 --no-pager (I chose not to show due to privacy) shows I'm having some fatal errors. However I'm not too sure what to make of those logs. However it looks like there is some issue with the fact that the network does not exist before trying to spin up those services.

There are no logs, docker logs <container_id>, in either containers which do not spin up.

This never happens if I stop the service and redeploy inside of docker.

I have tried following:

• restart: always
• depends_on
• Health checks

I really wish I don't have to SSH into the server and restart the containers manually every time the server restarts. It does a forceful restart every time we reconfigure the resources on the server.

Provide environment information

Operating System:
  OS: Ubuntu 22.04
  Arch: AMD64
Dokploy version: 0.26.1
VPS Provider: Glesys
Deploying: Compose with some services.

Which area(s) are affected? (Select all that apply)

Docker, Traefik, Docker Compose, Swarm

Are you deploying the applications where Dokploy is installed or on a remote server?

Remote server

Additional context

Due to security reason I cannot share any docker composes nor logs for now.

Will you send a PR to fix it?

Maybe, need help

[djsisson]

one thing to note, is in swarm, services do not have a depends on, so if upon restart docker brings dokploy up before the db or redis is available, then usually the container would error, exit and swarm would recreate and would keep doing so till its dependencies are available, providing it does not have some kind of check loop in its entrypoint.

however it is essential the container does give a bad exit code and exits in order to get restarted by swarm.

so it is not unusual to see exited services in swarm upon restart as long as they are healthy in the end.
but without logs its hard to say what else is going on, as networks are loaded before containers

also you mention adding services to the dokploy-network, this network is an overlay network, so swarm can handle container lifecycles for services on this network, if you are adding a normal compose container to this network, although you can do it, i think this will give you the error you see, since the daemon will try and start the container before swarm has finished initialising, so the overlay isn't available.

overlay networks are really only meant for multi-node swarm services, using it for a single node just adds network overhead

[MrTengil]

one thing to note, is in swarm, services do not have a depends on, so if upon restart docker brings dokploy up before the db or redis is available, then usually the container would error, exit and swarm would recreate and would keep doing so till its dependencies are available, providing it does not have some kind of check loop in its entrypoint.

however it is essential the container does give a bad exit code and exits in order to get restarted by swarm.

so it is not unusual to see exited services in swarm upon restart as long as they are healthy in the end. but without logs its hard to say what else is going on, as networks are loaded before containers

also you mention adding services to the dokploy-network, this network is an overlay network, so swarm can handle container lifecycles for services on this network, if you are adding a normal compose container to this network, although you can do it, i think this will give you the error you see, since the daemon will try and start the container before swarm has finished initialising, so the overlay isn't available.

overlay networks are really only meant for multi-node swarm services, using it for a single node just adds network overhead

Given this what are your recommendations? A pr which fixes this or change in my setup?

[djsisson]

afaik the swarm orchestrator will wait till the overlay networks are up before recreating the services, however its possible the daemon will try and start a normal container that's attached to an overlay network before the overlay network is initialized, but i thought as long as restart: always is set it would keep trying even with a network error.

without seeing the actual error i don't know what to suggest as i haven't seen this before, not a 255 error anyway, even with swarm services

[Obed0101]

+1

[Siumauricio]

Thanks for the detailed report! After investigating this, the behavior you're seeing is actually expected Docker Swarm behavior, not a Dokploy bug.

For Swarm services (dokploy, postgres, redis): When a forced shutdown happens, Swarm recreates the services as new containers on boot. The "Exited (255)" containers are the old ones — the new "Up" containers are their replacements. This is how Swarm is designed to work.

For docker-compose services connected to dokploy-network: The dokploy-network is an overlay network. After a forced reboot, there's a race condition at the Docker level — Docker tries to restart your containers before Swarm has fully initialized the overlay network, causing the exit code 255. This isn't something Dokploy controls( instead you can use isolated deployment)

Workarounds:

Add a systemd service or cron job that runs after Docker starts to clean up exited containers:

1. docker restart $(docker ps -qa -f "status=exited")
   Avoid forced shutdowns when possible
2. Clean up stale containers with docker container prune

As @djsisson mentioned, overlay networks on single-node setups add overhead and can cause these race conditions. This is a Docker/Swarm limitation, not a Dokploy issue.

Closing this as it's outside of Dokploy's scope, but feel free to reopen if you have further questions.