How to handle multiple scans from same date

[equant]

The image below is a screensnippet from Here

The issue is that there are three tarballs for the same date (and expirement: radish). This results in distributed_pipeline_wrapper.py not knowing what to do...

> /xdisk/dukepauli/equant/automation/distributed_pipeline_wrapper.py(181)get_irods_input_path()
-> raise ValueError(f"Could not find appropriate tarball for date: {date}\n \
(Pdb) matching_files
['scanner3DTop-2021-03-30__13-27-26-751_radish.tar.gz', 
'scanner3DTop-2021-03-30__13-38-29-051_radish.tar.gz', 
'scanner3DTop-2021-03-30__19-37-44-934_radish.tar.gz']

Obviously we don't care about the radish scans much, but can this happen with data we do care about? What if a scan starts, aborts and restarts? Would we see something like this?

If this could be an issue, how should we handle it?

• Should we error out with an explanation
• Should we take the largest file?
• How do we tell distributed_pipeline_wrapper.py which one to move forward with?
• Do we add a flag for the full filename to use in case of multiple scans for a given date?

[emmanuelgonz]

@equant The presence of multiple tarballs of the same species and date indicate some error during scanning from which scanning was then resumed. To account for this, we may want to consider adding the timestamp to our outputs so that we can prevent any overwriting on our CyVerse Data Store.