pybids shipped/used by 20.2.1 LTS is still curious about all things .git/

[yarikoptic]

What version of fMRIPrep are you using?

20.2.1-LTS

What kind of installation are you using? Containers (Singularity, Docker), or "bare-metal"?

I believe a container (dunno details)

crashes with

OSError: Error occurred while trying to decode JSON from file '/SENSORED/.git/annex/transfer/download/590b4fd0-0142-4e9d-8964-d1158c242c6a/MD5E-s102739--2349cac048f4ccf45bcb15278a783d26.json'.

within pybids -- not sure why to look under .git/ at all.

here is the traceback user provided

/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/validation.py:46: UserWarning: The ability to pass arguments to BIDS

Layout that control indexing is likely to be removed in future; possibly as early as PyBIDS 0.14. This includes the `config_filen

ame`, `ignore`, `force_index`, and `index_metadata` arguments. The recommended usage pattern is to initialize a new BIDSLayoutInd

exer with these arguments, and pass it to the BIDSLayout via the `indexer` argument.

  warnings.warn("The ability to pass arguments to BIDSLayout that control "

Traceback (most recent call last):

  File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/index.py", line 280, in _index_metadata

    payload = json.load(handle)

  File "/usr/local/miniconda/lib/python3.7/json/__init__.py", line 296, in load

    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)

  File "/usr/local/miniconda/lib/python3.7/json/__init__.py", line 348, in loads

    return _default_decoder.decode(s)

  File "/usr/local/miniconda/lib/python3.7/json/decoder.py", line 340, in decode

    raise JSONDecodeError("Extra data", s, end)

json.decoder.JSONDecodeError: Extra data: line 1 column 18 (char 17)



The above exception was the direct cause of the following exception:


Traceback (most recent call last):

  File "/usr/local/miniconda/bin/fmriprep", line 10, in <module>

    sys.exit(main())

  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py", line 17, in main

    parse_args()

  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/parser.py", line 602, in parse_args

    config.from_dict(vars(opts))

  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/config.py", line 617, in from_dict

    execution.load(settings)

  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/config.py", line 218, in load

    cls.init()

  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/config.py", line 453, in init

    re.compile(r"^\."),

  File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/layout.py", line 155, in __init__

    indexer(self)

  File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/index.py", line 111, in __call__

    self._index_metadata()

  File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/index.py", line 284, in _index_metadata

    raise IOError(msg) from e

OSError: Error occurred while trying to decode JSON from file '/SENSORED/.git/annex/transfer/download/590b4fd0-0142-4e9d-8964-d1158c242c6a/MD5E-s102739--2349cac048f4ccf45bcb15278a783d26.json'.

didn't try yet to dig deeper, but may be something (like "aha -- was addressed in pybids X which isn't included in this LTS, or didn't make it into LTS) rings a bell?

[yarikoptic]

@effigies et al -- any guidance to guide a user would be appreciated (insofar it was datalad clean && rm -rf .heudiconv)

[effigies]

We may be able to specifically ignore these things, since we'll have a pinned pybids in the lts.

But thanks for looking into this. Have a bit of a backlog...

[effigies]

Trying to reproduce inside a docker container fails. Are you able to provide a minimal reproduction?

[yarikoptic]

will try later on, meanwhile received another user's report with

>    OSError: Error occurred while trying to decode JSON from file
>    '/dartfs-hpc/rc/lab/XXXX/.heudiconv/sid00YYYY/info/filegroup.json'.

[oesteban]

Hey @yarikoptic, did you finally find the issue or further info for @effigies to replicate?

[yarikoptic]

will try to provide a complete reproducer today but I guess any broken symlink'ed .json under .heudiconv or .git/annex would trip it.

[yarikoptic]

I think it might have been resolved as of 20.2.3

here is a quick and dirty dump from bash history on how I reproduced first in 20.2.1:

mkdir reproducers
cd reproducers
datalad install ///openneuro/ds000001 && cd ds000001
mkdir -p .git/annex/transfer/download/
ln -s heavens.json .git/annex/transfer/download/inferno.json
ls -l .git/annex/transfer/download/inferno.json
mkdir -p .heudiconv/01/info/
ln -s heavens.json .heudiconv/01/info/wishful.json
datalad get -J4 sub-01

so I got a "random" (first) dataset from openneuro, created broken links to .json files in locations implicated in the issue report, and then (since still lacking space to docker pull the beast even after I freed up over 40GB on /, might be more to it) just used singularity container to reproduce one of the crashes using

mkdir tmp
singularity run -B $PWD:$PWD /home/yoh/proj/repronim/containers/images/bids/bids-fmriprep--20.1.1.sing $PWD $PWD/tmp/out participant --participant-label 01 -w $PWD/tmp/

and now running 20.2.3 using

cd ..
mkdir tmp
rm -rf ds000001/tmp
mkdir licenses
cp ~/.freesurfer-license licenses/
singularity run -B $PWD:$PWD /home/yoh/proj/repronim/containers/images/bids/bids-fmriprep--20.2.3.sing $PWD/ds000001 $PWD/tmp/out participant --participant-label 01 -w $PWD/tmp/ --fs-license-file $PWD/licenses/.freesurfer-license

and it looks like doing useful stuff now (might leave this session open overnight to see it complete), but staying positive, let's consider this resolved for now.

edit: finished without a fail -- great!