[ENH] Workflow for zipping a raw folder content

[marcobarilari]

I am writing a script to zip 4D brains in a fresh bidsifyed raw folder. To do so, I am taking inspiration from bidsCopyRawFolder ( I guess it will be handy in the near future to also understand how it works ;) ).

When the func. unzip the files, it uses load_untouch_nii and save_untouch_nii, what is the reason to do that instead of just using gunzip? Should I worry about any data format treatment when I gzip them?

Thanks :)

[Remi-Gau]

Actually, if you have a bidsified dataset then I think you can rely on spm_bids('query', 'data'...) to get all the *.nii files of the dataset and then loop through them to zip them.

I can't remember the reason why we decided to unzip things that way.

Actually I am tempted to do all the zipping and unzipping through SPM batches (in the batch editor: Basic I/O --> File/Dir operations --> File operations) because this way:

• we can ditch one of our depenency (nifti tools)
• using SPM ensures Octave compatibility
• and... Yeah you guessed it: going through a batch allows us to write tests.

[Remi-Gau]

Actually we might want to have a similar simple workflow to deface all the anat files in a dataset.

[marcobarilari]

Actually we might want to have a similar simple workflow to deface all the anat files in a dataset.

yes! this is the next one

[CerenB]

it's related but not directly under this issue, I can delete it if you see somewhere else is a better fit.
I got used to load_nii (nifti tools) and now in a script I got from a collaborator, it's with spm_vol, spm_read_vols, and some nifti tools functions such as load_untouch_nii and save_nii. I think I can replace either spm func or nifti func with one another.
would you recommend to ditch one of those? if so, which one?

[CerenB]

@marcobarilari out of curiosity, can I ask why do you need to zip them?

[marcobarilari]

Short answer: to save space (every 6GB you save ~1.5GB).

Long answer: we get the nii files from liege in single volumes, we bidsify them with bidsme and the result is a raw folder with not zipped files which is very large. Since we move around the data a lot (e.g. on gin), zipping them we save a lot of space and time

[Remi-Gau]

Short answer: to save space (every 6GB you save ~1.5GB).

Long answer: we get the nii files from liege in single volumes, we bidsify them with bidsme and the result is a raw folder with not zipped files which is very large. Since we move around the data a lot (e.g. on gin), zipping them we save a lot of space and time

#HighResMriKillsHardDrives

[Remi-Gau]

it's related but not directly under this issue, I can delete it if you see somewhere else is a better fit.
I got used to load_nii (nifti tools) and now in a script I got from a collaborator, it's with spm_vol, spm_read_vols, and some nifti tools functions such as load_untouch_nii and save_nii. I think I can replace either spm func or nifti func with one another.
would you recommend to ditch one of those? if so, which one?

Well... It depends.

If you do everything in an SPM context I would tend to use only SPM function.

If the script does not imply that you HAVE to use SPM then I would go for nifti tools as it is a lighter toolbox than SPM (though I am not sure it is Octave compatible).

Other issue to consider, not sure what number format is used by nifti tools to write data but it might be worth checking you do not get different results in terms of values between SPM and nifti tools.

So in general I would not recommend mixing things in the same pipeline.

[marcobarilari]

Actually, if you have a bidsified dataset then I think you can rely on spm_bids('query', 'data'...) to get all the *.nii files of the dataset and then loop through them to zip them.

This line of code unzippedNiifiles = spm_select('FPListRec', rawDir, '^.*.nii$'); does the job very well, see below with a mock raw folder

unzippedNiifiles

unzippedNiifiles =

  12×125 char array

    '/Users/barilari/Desktop/data_temp/raw_temp/sub-pilot001/ses-002/anat/sub-pilot001_ses-002_run-1_T1w.nii                      '
    '/Users/barilari/Desktop/data_temp/raw_temp/sub-pilot001/ses-002/fmap/sub-pilot001_ses-002_run-1_magnitude1.nii               '
    '/Users/barilari/Desktop/data_temp/raw_temp/sub-pilot001/ses-002/fmap/sub-pilot001_ses-002_run-1_magnitude2.nii               '
    '/Users/barilari/Desktop/data_temp/raw_temp/sub-pilot001/ses-002/fmap/sub-pilot001_ses-002_run-1_phasediff.nii                '
    '/Users/barilari/Desktop/data_temp/raw_temp/sub-pilot001/ses-002/func/sub-pilot001_ses-002_task-bimotionGrating_run-1_bold.nii'
    '/Users/barilari/Desktop/data_temp/raw_temp/sub-pilot001/ses-002/func/sub-pilot001_ses-002_task-bimotionGrating_run-2_bold.nii'
    '/Users/barilari/Desktop/data_temp/raw_temp/sub-pilot002/ses-002/anat/sub-pilot001_ses-002_run-1_T1w.nii                      '
    '/Users/barilari/Desktop/data_temp/raw_temp/sub-pilot002/ses-002/fmap/sub-pilot001_ses-002_run-1_magnitude1.nii               '
    '/Users/barilari/Desktop/data_temp/raw_temp/sub-pilot002/ses-002/fmap/sub-pilot001_ses-002_run-1_magnitude2.nii               '
    '/Users/barilari/Desktop/data_temp/raw_temp/sub-pilot002/ses-002/fmap/sub-pilot001_ses-002_run-1_phasediff.nii                '
    '/Users/barilari/Desktop/data_temp/raw_temp/sub-pilot002/ses-002/func/sub-pilot001_ses-002_task-bimotionGrating_run-1_bold.nii'
    '/Users/barilari/Desktop/data_temp/raw_temp/sub-pilot002/ses-002/func/sub-pilot001_ses-002_task-bimotionGrating_run-2_bold.nii'

also no need to loop cause in the batch you can set that the gzipped file will be save in the same folder of the unzipped one (and at the moment I don't see reason to modify this and give the option to save them in a different folder)

I can't remember the reason why we decided to unzip things that way.

Actually I am tempted to do all the zipping and unzipping through SPM batches (in the batch editor: Basic I/O --> File/Dir operations --> File operations) because this way:

• we can ditch one of our depenency (nifti tools)
• using SPM ensures Octave compatibility
• and... Yeah you guessed it: going through a batch allows us to write tests.

Ok cool! thanks, will use spm then

[marcobarilari]

To be clear, this func will not be so smart to get input on which e.g. modality/task skipping during the gzip process. So no really need I think to use query and retrieve other info than the path to all the .nii files