collate_meta_tensor to handle varying metadata keys

[wyli]

the collate_meta_tensor could be more permissive, if keys in the metadata are missing. I work with data from multiple datasets and frequently get errors, as keys are not the same across the different images. I usually just pop these keys in the metadata, but it would be nice, if this would not be an error at all.

  File "/home/bressekk/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/bressekk/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 61, in fetch
    return self.collate_fn(data)
  File "/home/bressekk/miniconda3/envs/monailabel/lib/python3.9/site-packages/monai/data/utils.py", line 470, in list_data_collate
    ret = collate_meta_tensor(data)
  File "/home/bressekk/miniconda3/envs/monailabel/lib/python3.9/site-packages/monai/data/utils.py", line 439, in collate_meta_tensor
    return {k: collate_meta_tensor([d[k] for d in batch]) for k in elem_0}
  File "/home/bressekk/miniconda3/envs/monailabel/lib/python3.9/site-packages/monai/data/utils.py", line 439, in <dictcomp>
    return {k: collate_meta_tensor([d[k] for d in batch]) for k in elem_0}
  File "/home/bressekk/miniconda3/envs/monailabel/lib/python3.9/site-packages/monai/data/utils.py", line 434, in collate_meta_tensor
    collated.meta = default_collate([i.meta or TraceKeys.NONE for i in batch])
  File "/home/bressekk/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 265, in default_collate
    return collate(batch, collate_fn_map=default_collate_fn_map)
  File "/home/bressekk/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 128, in collate
    return elem_type({key: collate([d[key] for d in batch], collate_fn_map=collate_fn_map) for key in elem})
  File "/home/bressekk/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 128, in <dictcomp>
    return elem_type({key: collate([d[key] for d in batch], collate_fn_map=collate_fn_map) for key in elem})
  File "/home/bressekk/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 128, in <listcomp>
    return elem_type({key: collate([d[key] for d in batch], collate_fn_map=collate_fn_map) for key in elem})
KeyError: 'ITK_non_uniform_sampling_deviation'

Originally posted by @kbressem in #5917 (comment)

[kbressem]

Thank you so much for raising this issue. I am currently bypassing this problem with some custom transforms, that manipulate the metadata (meta.py.zip). If these should be interesting for monai, I am happy to contribute them to the transforms.

[KumoLiu]

Looks like this PR can fix this issue.

[Sere1nz]

i still met similar issue, what should I do to avoid it?

epoch 1/300
1/103, train_loss: 0.6715
loss: 0.6715
2/103, train_loss: 0.6605
loss: 0.6660
3/103, train_loss: 0.6443
loss: 0.6588
4/103, train_loss: 0.6292
loss: 0.6514
5/103, train_loss: 0.6152
loss: 0.6441
6/103, train_loss: 0.6008
loss: 0.6369
7/103, train_loss: 0.5907
loss: 0.6303
8/103, train_loss: 0.5750
loss: 0.6234
9/103, train_loss: 0.5689
loss: 0.6173
10/103, train_loss: 0.5599
loss: 0.6116
Traceback (most recent call last):
  File "/mnt/d/monai_tutorial/nodule_3d_seg/swinunetr/train.py", line 156, in <module>
    for batch_data in train_loader:
  File "/home/zzm/anaconda3/envs/monai-nnunet/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 630, in __next__
    data = self._next_data()
  File "/home/zzm/anaconda3/envs/monai-nnunet/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 674, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/zzm/anaconda3/envs/monai-nnunet/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 54, in fetch
    return self.collate_fn(data)
  File "/home/zzm/anaconda3/envs/monai-nnunet/lib/python3.9/site-packages/monai/data/utils.py", line 670, in pad_list_data_collate
    return PadListDataCollate(method=method, mode=mode, **kwargs)(batch)
  File "/home/zzm/anaconda3/envs/monai-nnunet/lib/python3.9/site-packages/monai/transforms/croppad/batch.py", line 114, in __call__
    return list_data_collate(batch)
  File "/home/zzm/anaconda3/envs/monai-nnunet/lib/python3.9/site-packages/monai/data/utils.py", line 490, in list_data_collate
    ret = collate_meta_tensor(data)
  File "/home/zzm/anaconda3/envs/monai-nnunet/lib/python3.9/site-packages/monai/data/utils.py", line 459, in collate_meta_tensor
    return {k: collate_meta_tensor([d[k] for d in batch]) for k in elem_0}
  File "/home/zzm/anaconda3/envs/monai-nnunet/lib/python3.9/site-packages/monai/data/utils.py", line 459, in <dictcomp>
    return {k: collate_meta_tensor([d[k] for d in batch]) for k in elem_0}
  File "/home/zzm/anaconda3/envs/monai-nnunet/lib/python3.9/site-packages/monai/data/utils.py", line 454, in collate_meta_tensor
    collated.meta = default_collate([i.meta or TraceKeys.NONE for i in batch])
  File "/home/zzm/anaconda3/envs/monai-nnunet/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 265, in default_collate
    return collate(batch, collate_fn_map=default_collate_fn_map)
  File "/home/zzm/anaconda3/envs/monai-nnunet/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 127, in collate
    return elem_type({key: collate([d[key] for d in batch], collate_fn_map=collate_fn_map) for key in elem})
  File "/home/zzm/anaconda3/envs/monai-nnunet/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 127, in <dictcomp>
    return elem_type({key: collate([d[key] for d in batch], collate_fn_map=collate_fn_map) for key in elem})
  File "/home/zzm/anaconda3/envs/monai-nnunet/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 127, in <listcomp>
    return elem_type({key: collate([d[key] for d in batch], collate_fn_map=collate_fn_map) for key in elem})
KeyError: 'MITK_IO_reader_description'

Data was converted from dicom to nii.gz. for convenience. Except this key error, sometimes I met other KeyError: 'DICOM_0008_...'

[Sere1nz]

epoch_ 1/300
Traceback (most recent call last):
  File "/mnt/d/monai_tutorial/nodule_3d_seg/swinunetr/train.py", line 155, in <module>
    for batch_data in train_loader:
  File "/home/zzm/anaconda3/envs/monai-nnunet/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 630, in __next__
    data = self._next_data()
  File "/home/zzm/anaconda3/envs/monai-nnunet/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 674, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/zzm/anaconda3/envs/monai-nnunet/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 54, in fetch
    return self.collate_fn(data)
  File "/home/zzm/anaconda3/envs/monai-nnunet/lib/python3.9/site-packages/monai/data/utils.py", line 490, in list_data_collate
    ret = collate_meta_tensor(data)
  File "/home/zzm/anaconda3/envs/monai-nnunet/lib/python3.9/site-packages/monai/data/utils.py", line 459, in collate_meta_tensor
    return {k: collate_meta_tensor([d[k] for d in batch]) for k in elem_0}
  File "/home/zzm/anaconda3/envs/monai-nnunet/lib/python3.9/site-packages/monai/data/utils.py", line 459, in <dictcomp>
    return {k: collate_meta_tensor([d[k] for d in batch]) for k in elem_0}
  File "/home/zzm/anaconda3/envs/monai-nnunet/lib/python3.9/site-packages/monai/data/utils.py", line 454, in collate_meta_tensor
    collated.meta = default_collate([i.meta or TraceKeys.NONE for i in batch])
  File "/home/zzm/anaconda3/envs/monai-nnunet/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 265, in default_collate
    return collate(batch, collate_fn_map=default_collate_fn_map)
  File "/home/zzm/anaconda3/envs/monai-nnunet/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 127, in collate
    return elem_type({key: collate([d[key] for d in batch], collate_fn_map=collate_fn_map) for key in elem})
  File "/home/zzm/anaconda3/envs/monai-nnunet/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 127, in <dictcomp>
    return elem_type({key: collate([d[key] for d in batch], collate_fn_map=collate_fn_map) for key in elem})
  File "/home/zzm/anaconda3/envs/monai-nnunet/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 127, in <listcomp>
    return elem_type({key: collate([d[key] for d in batch], collate_fn_map=collate_fn_map) for key in elem})
KeyError: 'DICOM_0008_0018'

[KumoLiu]

Hi @Sere1nz, what's your MONAI version?
This PR has been merged in 1.3. Could you please try monai1.3? Thanks!

[Sere1nz]

@KumoLiu I used monai 1.2 on the dataset on remote ubuntu server without issue.
I met this problem when using monai1.2 on my laptop on the exact same dataset as on the remote server.
I upgrade monai to 1.3 and it works well now! It is weird.
Anyway, thank you!