FileNotFoundError: [Errno 2] No such file or directory: '_task-rest_bold.json'

[m-petersen]

Summary

I try to implement bids conversion with heudiconv on our local HPC using a singularity container ( heudiconv-0.9.0). As I will work with a large cohort (>2000 subjects) I am currently implementing a parallelization across nodes via SLURM and within each node with GNU parallel with a test dataset (5 times the same subject). In doing so, a test run fails with the following error.

`local:5/0/100%/0.0s 0: Traceback (most recent call last):
0:   File "/opt/miniconda-latest/bin/heudiconv", line 33, in <module>
0:     sys.exit(load_entry_point('heudiconv', 'console_scripts', 'heudiconv')())
0:   File "/src/heudiconv/heudiconv/cli/run.py", line 24, in main
0:     workflow(**kwargs)
0:   File "/src/heudiconv/heudiconv/main.py", line 351, in workflow
0:     grouping=grouping,)
0:   File "/src/heudiconv/heudiconv/convert.py", line 238, in prep_conversion
0:     getattr(heuristic, 'DEFAULT_FIELDS', {}))
0:   File "/src/heudiconv/heudiconv/bids.py", line 94, in populate_bids_templates
0:     populate_aggregated_jsons(path)
0:   File "/src/heudiconv/heudiconv/bids.py", line 126, in populate_aggregated_jsons
0:     json_ = load_json(fpath)
0:   File "/src/heudiconv/heudiconv/utils.py", line 177, in load_json
0:     with open(filename, 'r') as fp:
0: FileNotFoundError: [Errno 2] No such file or directory: '/bids/sub-ewgenia001/ses-1/func/sub-ewgenia001_ses-1_task-rest_bold.json'` 

Interestingly this affects only 4 of 5 subjects with the remaining one (seemingly always the first subject completing) without issues. All this sounds a little bit like race-condition as discussed in #362. However, as far as I understand a fix has been implemented and I am working with datalad creating a ephemeral clone of the dataset on a scratch partition for each subject before applying heudiconv to it (scripts below). Maybe I am misunderstanding something but shouldn't the latter somehow address the race condition with the processes writing to a separate/cloned top-level file?

Any input would be highly appreciated. Happy to provide further details.

heudiconv_heuristic.txt
pipelines_parallelization.txt (batch script parallelizing pipelines_processing across subjects with GNU parallel)
pipelines_processing.txt

Platform details:

Choose one:

• [ x] Container (heudiconv 0.9.0)

[m-petersen]

To follow up, using --bids notop heudiconv runs without issues.

[yarikoptic]

ha -- situation is a bit different from #362 -- I guess while the populate_aggregated_jsons (locked) is going through the long list of .json files it got, some other parallel process managed to remove that file (may be to just recreate with updated content or smth like that). Do not see a clear way out yet besides making some load_json_wait to be used there which would also wait for the file to re-appear in some reasonable duration of time, so smth along the lines of

diff --git a/heudiconv/utils.py b/heudiconv/utils.py
index f30a23e..7384cf7 100644
--- a/heudiconv/utils.py
+++ b/heudiconv/utils.py
@@ -14,6 +14,7 @@ from collections import namedtuple
 from glob import glob
 from subprocess import check_output
 from datetime import datetime
+from time import sleep
 
 from nipype.utils.filemanip import which
 
@@ -173,12 +174,17 @@ def load_json(filename):
     -------
     data : dict
     """
-    try:
-        with open(filename, 'r') as fp:
-            data = json.load(fp)
-    except JSONDecodeError:
-        lgr.error("{fname} is not a valid json file".format(fname=filename))
-        raise
+    for i in range(100):  # >= 10 sec wait
+        try:
+            with open(filename, 'r') as fp:
+                data = json.load(fp)
+                break
+        except JSONDecodeError:
+            lgr.error("{fname} is not a valid json file".format(fname=filename))
+            raise
+        except FileNotFoundError:
+            sleep(0.1)
+            continue
 
     return data
     

but I guess it is not something you could try out easily right?

[yarikoptic]

I thought to suggest meanwhile that you could run all individual conversions indeed with notop but then "conclude" with a single run of --command populate-templates but I saw that add_participant_record is not ran then, so you would end up with a not filled out participants.tsv :-/

[m-petersen]

Hi Yaroslav,

thanks for your help.

but I guess it is not something you could try out easily right?

Indeed, it isn't something I can try easily with my setup. Nevertheless, --bids notop works fine and I fill the participants.tsv afterwards.

[burdinskid13]

I ran into the same issue as the original poster @m-petersen -- has any solution for this been implemented yet?

[mgxd]

@burdinskid13 it looks like this bug is still around - the best workaround for now seems to be #516 (comment)

[yarikoptic]

sorry for the delay. I have now implemented that workaround in the comment as #523 . I think it should be safe, rapid review would be appreciated. If no objections etc, I will merge tomorrow and kick out a fresh heudiconv version -- it has been awhile

[github-actions]

🚀 Issue was released in v0.11.1 🚀

[yarikoptic]

sorry - referenced this issue incorrectly within #564, so it was released some time before (I guess 0.10.0)