Logger collector includes test results from other dataloaders

[simbara]

🐛 Bug

Logger collection in test step does not work correctly when number of test loaders is 10 or greater

To Reproduce

Collab link: https://colab.research.google.com/drive/1iNhae2jTUBXL4knJz5hd3CvfrY5D_bTD?usp=sharing

Expected behavior

DATALOADER:1 TEST RESULTS should only include results from dataloader_idx_1

DATALOADER:0 TEST RESULTS
{'fake_test_acc/dataloader_idx_0': 9.876543925421439e-15}
--------------------------------------------------------------------------------
DATALOADER:1 TEST RESULTS
{'fake_test_acc/dataloader_idx_1': 9.876543925421439e-15,
 'fake_test_acc/dataloader_idx_10': 9.876543925421439e-15,
 'fake_test_acc/dataloader_idx_11': 9.876543925421439e-15,
 'fake_test_acc/dataloader_idx_12': 9.876543925421439e-15,
 'fake_test_acc/dataloader_idx_13': 9.876543925421439e-15,
 'fake_test_acc/dataloader_idx_14': 9.876543925421439e-15,
 'fake_test_acc/dataloader_idx_15': 9.876543925421439e-15,
 'fake_test_acc/dataloader_idx_16': 9.876543925421439e-15,
 'fake_test_acc/dataloader_idx_17': 9.876543925421439e-15,
 'fake_test_acc/dataloader_idx_18': 9.876543925421439e-15,
 'fake_test_acc/dataloader_idx_19': 9.876543925421439e-15}
--------------------------------------------------------------------------------
DATALOADER:2 TEST RESULTS
{'fake_test_acc/dataloader_idx_2': 9.876543925421439e-15}

Environment

• CUDA:
  • GPU:
    • Tesla K80
  • available: True
  • version: 11.1
• Packages:
  • numpy: 1.19.5
  • pyTorch_debug: False
  • pyTorch_version: 1.10.0+cu111
  • pytorch-lightning: 1.5.1
  • tqdm: 4.62.3
• System:
  • OS: Linux
  • architecture:
    • 64bit
  • processor: x86_64
  • python: 3.7.12
  • version: Proposal for help #1 SMP Sat Jun 5 09:50:34 PDT 2021

Additional context

https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/trainer/connectors/logger_connector/logger_connector.py#L164-L166

 k: v for k, v in metrics.items() if "dataloader_idx" not in k or f"dataloader_idx_{dl_idx}" in k

f"dataloader_idx_{dl_idx}" in k

will match all substrings

[rohitgr7]

nice catch! interested in sending a PR with the fix? :)

[shabie]

I could take this up if it's still for available for grabs :)

[rohitgr7]

yes please, go ahead 🚀