incremental mode of operation

[yarikoptic]

Description

We would like to introduce mrQA to our "pipeline" of data acquisition. I wish there was a mode where mrQA could have been used on DICOMs in an incremental fashion. E.g. we collect 1st subject/session, run mrQA which extracts all metadata etc, stores it and produces report on what it can tell up to that point. Then for the next subject/session we point mrQA to prior save extracts and that new acqusition(s) so it could update extracts and the report.

Let me know if you need me to elaborate more on this.

[raamana]

hi Yarik, that is exactly what had been working on now for the past few months and we call it monitoring and it's on the branch https://github.com/Open-Minds-Lab/mrQA/tree/mrqa_monitor that will soon merge to main. Do you wanna take a look and test it? Thank you.

@sinhaharsh

[raamana]

So the current design focusses on simply updating the existing compliance report, but we could also produce deltas in terms of what subjects and sessions have changed, and what in the report has been changed. I personally don't think the Delta in the report is not very useful, but you may have other uses for it, and it's rather not that difficult to produce. Let me know. Thanks.

[raamana]

docs: https://open-minds-lab.github.io/mrQA/examples.html#monitoring-use-case

[yarikoptic]

Great to hear that you are working on pretty much what we need! ;) One tricky aspect is that, according to the doc you point out to, you depend on a singular --data-source /path/to/dataset . In our case we have accessions across all studies falling under some common /path/to/dataset and we would like to give specific new (can do with old ones if needed) subdirectories for monitoring/update. E.g. it would be smth like we have D1, D2, D3, etc folders but want to include in the QA only D1, D3, D10 ... which we figure out what to include based on some metadata within them. Do you think it would be possible or complicate situation too much?

BTW -- can it be .tgz archives with dicoms? then we could point to sourcedata/ of the converted BIDS dataset (e.g. http://datasets.datalad.org/?dir=/dbic/QA/sourcedata) but it would require first running conversion and I wanted to add mrQA as early in the pipeline as "process all dicoms received today and adjust reports for each of the corresponding studies", possibly before we get to BIDS conversion. Well, thinking about it -- would lead to situation where study decides to drop specific accession etc, which would get reflected while converting to BIDS but might not be known at the "DICOMs level".. we will see...

[raamana]

1. there is no specific constraints on /path/to/dataset - it could be anything you want as long as there are valid DICOM files inside. You can setup a cron job on top of this however you like (exposing to different datasets as is convenient for you)

2. we chatted about this in another issue in - see support some kind of dicom-archives style? #22 (comment)
   we are yet to work on it, and happy to take your help.