Cannot init, destroy, and then re-init process groups

[rohan-varma]

🐛 Bug

Calling dist.init_process_group followed by c10d.destroy_process_group and then re-initing pg does not appear to work, hanging in _store_based_barrier. See the following repro:

@requires_nccl()
    @skip_if_lt_x_gpu(2)
    def test_init_then_reinit_pg(self):
        store = c10d.FileStore(self.file_name, self.world_size)
        # process_group = c10d.ProcessGroupNCCL(store, self.rank, self.world_size)
        print("initializing pg")
        dist.init_process_group(backend="nccl", world_size=self.world_size, rank=self.rank, store=store)
        print("Done init pg")
        c10d.destroy_process_group()
        import time ; time.sleep(5)
        dist.init_process_group(backend="nccl", world_size=self.world_size, rank=self.rank, store=store)

The recently added python tracebacks indicate the issue is in _store_based_barrier: P402636997.

Specifically, it looks like destroy_process_group resets _group_count. This means that when we call init_process_group a 2nd time with the same store, it queries for a group count of 1, and finds it already in the store, changing the behavior of the worker counting logic in _store_based_barrier.

A workaround for now is using a new store each time. However, I thought that we already wrap stores with PrefixStore as needed to avoid these sort of collision issues.

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu @gcramer23

[rohan-varma]

cc @pritamdamania87 for store based barrier issue

[kit1980]

I've just saw a old TODO that depends on this. Was there any progress with the issue?