Upgrade from 3.6 to 3.7 RMQ

[harishankard]

When we do update/migrate from 3.6 to 3.7(3 nodes cluster), we get the mentioned below error . After all the nodes gets updated, sometimes the cluster never comes up.
2018-05-17 06:23:04.073 [error] <0.5.0>
Error description:
init:do_boot/3
init:start_em/1
rabbit:start_it/1 line 445
rabbit:'-boot/0-fun-0-'/0 line 296
rabbit_upgrade:run_mnesia_upgrades/2 line 155
rabbit_upgrade:die/2 line 212
throw:{upgrade_error,"\n\n****\n\nCluster upgrade needed but other disc nodes shut down after this one.\nPlease first start the last disc node to shut down.\n\nNote: if several disc nodes were shut down simultaneously they may all\nshow this message. In which case, remove the lock file on one of them and\nstart that node. The lock file on this node is:\n\n /var/vcap/store/rabbitmq/mnesia/db/nodes_running_at_shutdown \n\n****\n\n\n"}
Log file(s) (may contain more information):

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[bgandon]

In-place upgrades from RabbitMQ 3.6 to 3.7 are not supported. This was discussed recently here.

Please refer to this documentation chapter: Upgrading RabbitMQ Using Blue-Green Deployment Strategy, and this article by Pivotal RabbitMQ experts: Blue-Green Application Deployments with RabbitMQ.

[blgm]

Hi @harishankard, this is a limitation in the core RabbitMQ product. One approach is to progressively shut down all but one nodes in the cluster (allowing time for queue mirroring etc...), perform the upgrade, then scale the cluster back up. It it recommended to stop all apps and drain the queues first, otherwise the single node can become overloaded.

[blgm]

Hi @harishankard - We're closing this issue given we've provided a way forward and we're currently not looking at automating this process just yet. We would be interested in feedback on whether this works for you.

[harishankard]

Hi @blgm - When we used deprecated clutterer plugin, this error was not coming up.

[michaelklishin]

That's because rabbitmq-clusterer modifies the way nodes are started, so the error is effectively delayed. See Restarting Cluster Nodes and Cluster Upgrade doc guides.