Potential deadlock

[vineetgoel]

I noticed that when new boxes came up for scroll, it when into a heartbeat failure loop which is typical of deadlocks during the rebalance. I was able to confirm that there was a deadlock in the on_partition_revoked as can be seen here:

[2018-06-06 17:38:41,498: INFO]: [^-App]: [Flight Recorder-3] (started at Wed Jun  6 17:37:41 2018) Replaying logs...
[2018-06-06 17:38:41,499: INFO]: [^-App]: [Flight Recorder-3] (Wed Jun  6 17:37:41 2018) fetcher.stop()
[2018-06-06 17:38:41,499: INFO]: [^-App]: [Flight Recorder-3] (Wed Jun  6 17:37:41 2018) topics.on_partitions_revoked()
[2018-06-06 17:38:41,499: INFO]: [^-App]: [Flight Recorder-3] (Wed Jun  6 17:37:41 2018) tables.on_partitions_revoked()
[2018-06-06 17:38:41,499: INFO]: [^-App]: [Flight Recorder-3] (Wed Jun  6 17:37:41 2018) +TABLES: maybe_abort_ongoing_recovery
[2018-06-06 17:38:41,499: INFO]: [^-App]: [Flight Recorder-3] -End of log-

[vineetgoel]

This seems to be fixed now.