**gather_all_tensors_if_available** share the same underlying storage for all GPUs

[ShomyLiu]

🐛 Bug

Hi, one of new features in #2528 gather_all_tensors_if_available has a list copy bug, and this would lead that tensors in all GPUs are the wrongly same as one GPU, since they share the same storage:

https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/metrics/converters.py#L304

gathered_result = world_size * [torch.zeros_like(result)]

change into:

gathered_result = [torch.zeros_like(result) for _ in range(world_size)]

[edenlightning]

@justusschock @SkafteNicki

[SkafteNicki]

@ShomyLiu good catch, would you be up for sending a PR? Please, note that the function is not used anywhere yet, but are there for future changes to the metric package.

[ShomyLiu]

@SkafteNicki it's my pleasure for a PR. I will finish this as soon as possible.

Yeah, it's a new function to wrap the torch.distributed.all_gather. But I think it is a very common use case; especially, when using DDP mode, we always need to gather all the outputs cross all the GPUs.

[SkafteNicki]

@ShomyLiu Yes, I agree that it is a common use case.
Please ping me when PR is ready.

[ShomyLiu]

@SkafteNicki Hi, I have sent a PR jus now for your review #3319