Duplicate dynamic shovels can be started in certain scenarios

[michaelklishin]

Originally reported in #3132 and #3154 that @mkuratczyk and I could reproduce (hopefully reliably) using the Kubernetes cluster Operator.

The steps are:

• Form cluster S, the source
• Form cluster D, the destination
• Configure a Shovel in the D cluster so that it uses an exchange source in cluster S
• Verify that the shovel is running and functional
• For every node, put it under maintenance then restart
• Observe duplicate mirrored supervisor children with rabbitmqctl eval 'mirrored_supervisor:which_children(rabbit_shovel_dyn_worker_sup_sup).'

My testing suggests that just a restart is not enough but putting a node under maintenance mode first at least makes the issue
significantly more likely to resurface. Something like this:

for n in rabbit hare flopsy
do
  rabbitmq-upgrade drain -n $n@hostname
  rabbitmqctl stop_app -n $n@hostname
  rabbitmqctl start_app -n $n@hostname
done

[michaelklishin]

One discovered difference so far is that pg2 used to effectively use global locks for certain operations whereas pg is strongly eventually consistent, which mirrored_supervisor does not expect.

[michaelklishin]

The milestone says 3.8.20 but #3263 will also ship in 3.9.2.