Can't promote a node as manager on a swarm with docker nodes on 19.03.15

[vp-acouty]

Tell us about your request

I would like to upgrade my docker swarm cluster from 19 (Debian 9) to 23 (Debian 11).

Which service(s) is this request for?

Docker Engine

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

I'm trying to upgrade my docker swarm. I have 3 nodes on docker 19.03.15 (Debian 9)
When I add a node (Debian 11) on 23.0.1 as manager it appears. but as sooon as I promote it I loose the connectivity

Error message:

Error response from daemon: This node is not a swarm manager. Manager is being prepared or has trouble connecting to the cluster.

When I try to add the node as a manager directly

Error response from daemon: manager stopped: can't initialize raft node: rpc error: code = Unknown desc = could not connect to prospective new cluster member using its advertised address: rpc error: code = DeadlineExceeded desc = context deadline exceeded

Are you currently working around the issue?

Waiting to upgrade my cluster

Additional context

[vp-acouty]

This was a firewall issue. I had to open port 2377 to promote it.