Provide a basic function to load datalist from CSV file (25/June)

[Nic-Ma]

Is your feature request related to a problem? Please describe.
As we discussed in: #2212
We need to load CSV content and make it as a datalist for dataset.
Then apply transforms for the data items.

[Nic-Ma]

After discussion with @wyli , maybe it's better to involve the pandas lib to handle CSV for complicated features in the future. I will rewrite the draft code to use pandas DataFrame to support joining tables.
Please feel free to let me know if you have any concerns about optionally importing pandas.

Thanks.

[wyli]

Hi @Nic-Ma, the discussion outcome is to make some improvements to the current prototype as our basic module.

the current prototype for your reference:

data_train = MultimodalCSVDataset(csv_file='./test.csv',
                                  variable_groups = [['subject_id'],
                                                     ['label'],
                                                     ['image'],
                                                     ['image_second'],
                                                     ['image_full'],
                                                     ['ehr_%01d'%i for i in range(18)],
                                                     ['meta_%01d'%i for i in range(3)],
                                                     ],
                                  variable_group_names=['subject_id',
                                                        'label',
                                                        'image',
                                                        'image_full',
                                                        'ehr',
                                                        'flags'],
                                  variable_group_types=['str',
                                                        'int',
                                                        'str',
                                                        'str',
                                                        'np.float32',
                                                        'np.float32'],
                                  transform=transforms_train)

Enhancements TODOs

• the dataset should support single csv_file for the first version
• the dataset should support an additional dictionary for variable names, types, default for missing values,
  for example:

column_types = {  # this could come from some json-based mapping file https://www.w3.org/2013/csvw/wiki/CSV-LD)
    "subject_id": {"type": str}, 
    "label": {"type": int, "default": 0},
    "ehr_0": {"type": float, "default": 0.0, "unit": "year"},
    "ehr_1": {"type": float, "default": 0.0, "unit": "month"},
    "image": {"type": nifti, "default": None},
}
ds = MultimodalCSVDataset(
    csv_file='./test.csv',
    column_types=column_types,
    groups=[['subject_id'], ['label'], ['ehr_01', 'ehr_02'], ['image']],
    output_keys=['suject_id', 'label', 'ehr', 'img'],
    transform=transforms_train,
)

ds[0] would be a dictionary of

{"subject_id": "abcd", 
 "label": 1, 
 "ehr": [0.1, 0.2], 
 "img": "/Volume/file.nii.gz"}

cc @csudre @IntegratorBrad

the csv example:

,subject_id,label,image,image_full,ehr_0,ehr_1,ehr_2,ehr_3,ehr_4,ehr_5,ehr_6,ehr_7,ehr_8,ehr_9,ehr_10,ehr_11,ehr_12,ehr_13,ehr_14,ehr_15,ehr_16,ehr_17,meta_0,meta_1,meta_2
0,s000000,5,./imgs/s000000.png,/claraDevDay/Data/MultimodalMNIST/images_full/s000000.png,2.007843255996704,2.290196180343628,2.0549020767211914,3.0196080207824707,3.8078432083129883,3.5843138694763184,3.141176462173462,3.196078300476074,4.211764812469482,6.301961421966553,6.470588684082031,6.254902362823486,6.682353496551514,7.266666412353516,5.4509806632995605,0.0,0.0,0.0,True,True,True
1,s000001,0,./imgs/s000001.png,/claraDevDay/Data/MultimodalMNIST/images_full/s000001.png,6.8392157554626465,6.4745097160339355,5.8627448081970215,5.19215726852417,5.274509906768799,5.250980377197266,4.647058963775635,4.886274337768555,4.392156600952148,5.219608306884766,7.827450752258301,9.282353401184082,7.3176469802856445,4.266666889190674,0.0,0.0,0.0,0.0,False,True,False
2,s000002,4,./imgs/s000002.png,/claraDevDay/Data/MultimodalMNIST/images_full/s000002.png,3.7725489139556885,4.211764812469482,4.635294437408447,5.298039436340332,9.545097351074219,12.572548866271973,6.799999713897705,2.196078300476074,1.8823529481887817,1.8823529481887817,2.031372547149658,2.0352940559387207,2.2588233947753906,2.2627451419830322,1.9725490808486938,0.0,0.0,0.0,True,False,True
3,s000003,1,./imgs/s000003.png,/claraDevDay/Data/MultimodalMNIST/images_full/s000003.png,3.3333332538604736,3.2352943420410156,3.4000000953674316,3.164705753326416,3.0862746238708496,3.725490093231201,3.6980392932891846,3.6980390548706055,3.701960802078247,3.3098039627075195,3.729412078857422,3.6235294342041016,3.090195894241333,3.090195894241333,2.7058823108673096,0.0,0.0,0.0,False,False,True

[Nic-Ma]

Hi @wyli ,

Thanks for providing the reference examples.
I am updating the PR for this feature, the support of very big CSV file is challenging, I am checking other repos.

Thanks.

[Nic-Ma]

Hi @wyli ,

For this initial feature request, I think the PR #2349 already can support.
The load_csv_datalist() API can give a list of dictionaries, then use it in any Dataset, like our CacheDataset, PersistentDataset, etc. Same as our load_decathlon_datalist() API.
For example:

result = load_csv_datalist(
  filename=[filepath1, filepath2, filepath3],
  row_indices=[1, 3],  # load row: 1, 3
  col_names=["subject_id", "image", *[f"ehr_{i}" for i in range(11)], "meta_0", "meta_1", "meta_2"],
  col_groups={"ehr": [f"ehr_{i}" for i in range(11)], "meta12": ["meta_1", "meta_2"]},
  )
dataset = CacheDataset(data=result, transform=transforms, cache_rate=1.0)

The challenge is to support very big CSV files during training.
What do you think?

Thanks.

[Nic-Ma]

CC @ericspod @rijobro to join the discussion.

Thanks in advance.

[wyli]

the idea is to have a dataset class instead of a function, so that it'll be able to maintain internal states if needed and be extensible, similar to the imagedataset https://github.com/Project-MONAI/MONAI/blob/dev/monai/data/image_dataset.py#L23

[Nic-Ma]

OK, make sense.
Let me update it to a Dataset ASAP.

Thanks.

[ericspod]

What is the definition of very large CSV files? When they get to be hundreds of megabytes we start having issue with Pandas, @atbenmurray and others have been working on a project ExeTera for loading data from the ZOE Covid Symptoms app. It has its own way of parsing large CSV files because other things were painfully slow for that scale of data. I wouldn't necessarily recommend using this as it's not meant for online training, but it illustrates the challenges of large scale csv data. The alternative approach is of course to break files into smaller ones and cycle between them during training, unloading data from one before moving to the next. Pandas will be able to handle things at that scale so we can stick to pre-existing solutions.

[Nic-Ma]

Hi @ericspod ,

Thanks very much for your sharing!
In the initial PR #2349 , I tried to provide 2 Datasets, 1 is for regular small CSV files, 1 is for big CSV files and based on IterableDataset of PyTorch. So I think users can decide to use which Dataset based on his data and HW/SW environment.

Thanks.

[wyli]

see also https://github.com/Project-MONAI/MONAI/wiki/Minutes-Data-Working-Group-17-Jun-2021