Evaluation suite - loading input

[csudre]

Context
Evaluation suite - Need to be able to address multiple input situations - Either a single pair of reference / output or a folder of matching pairs or numpy arrays already loaded to memory.

Describe the solution you'd like
Flexible evaluation suite where input can be a specified pair of images to load / a folder with subject matching pairs
Should also be able to handle existing numpy arrays stored in memory

https://github.com/Project-MONAI/MONAI/wiki/Evaluation-metrics-task-force

[wyli]

thanks @csudre, for those three use cases, the basic evaluation logic would be:

1. user-specified pair of images to load:

datalist = [{'img': 'subject-1.nii.gz', 'seg': 'subject-1-seg.nii.gz'}]
dataset = monai.data.Dataset(datalist, transform=preprocessing)
loader = DataLoader(dataset, shuffle=False)
results = [compute_xx_metric(item) for item in loader]

2. user-specified folder:

data_folder = 'myfolder/'
datalist = folder_to_list(data_folder)
dataset = monai.data.Dataset(datalist, transform=preprocessing)
loader = DataLoader(dataset, shuffle=False)
results = [compute_xx_metric(item) for item in loader]

3. in memory arrays:

data_arrays = [{'img': some_nparray, 'seg': some_nparray}]
dataset = monai.data.Dataset(data_arrays, transform=preprocessing)
loader = DataLoader(dataset, shuffle=False)
results = [compute_xx_metric(item) for item in loader]

looks like what's missing just now is

• a folder_to_list utility to extract a data_list such as datalist = [{'img': 'subject-1.nii.gz', 'seg': 'subject-1-seg.nii.gz'}] from a user provided folder by listdir and match the filenames.
• a utility function to write results into a user-friendly format (csv?) or print results properly to the CLI
  what do you think @Nic-Ma @csudre