Provenance files appear to be put in the wrong dir during processing, resulting in FileNotFound error when copying to outdir

[grdryn]

Summary

When I try to use --with-prov, it fails to move/copy provenance file, seemingly because it's looking in the wrong place. It's expecting it to be in the temp dir that it creates, but it seems like the provenance file actually gets created in the directory that the command is run from (pwd).

Command used (run against the sample test data from this repo):

heudiconv --files /var/home/gryan/src/github.com/nipy/heudiconv/heudiconv/tests/data/ --outdir ../outgoing/ --heuristic reproin --bids --with-prov

Output:

INFO: Running heudiconv version 0.11.3 latest 0.11.3
INFO: Analyzing 11 dicoms
INFO: Filtering out 0 dicoms based on their filename
WARNING: Ignoring /var/home/gryan/src/github.com/nipy/heudiconv/heudiconv/tests/data/sample_nifti.nii.gz since not quite a "normal" DICOM: 'FileDataset' object has no attribute 'SeriesNumber'
WARNING: Ignoring /var/home/gryan/src/github.com/nipy/heudiconv/heudiconv/tests/data/sample_nifti_params.txt since not quite a "normal" DICOM: 'FileDataset' object has no attribute 'SeriesNumber'
INFO: Generated sequence info for 5 studies with 7 entries total
INFO: Processing sequence infos to deduce study/session
INFO: Study session for StudySessionInfo(locator='head/advanced applications libraries', session=None, subject='dbictest1')
INFO: Processing sequence infos to deduce study/session
INFO: Study session for StudySessionInfo(locator='Development/Functional', session=None, subject='o6jgilro')
INFO: Processing sequence infos to deduce study/session
INFO: Study session for StudySessionInfo(locator='Halchenko/Yarik/950_bids_test4', session='localizer', subject='phantom1sid1')
INFO: Processing sequence infos to deduce study/session
INFO: Study session for StudySessionInfo(locator='Research/MCBI_TESTING', session=None, subject='crlab')
INFO: Processing sequence infos to deduce study/session
INFO: Study session for StudySessionInfo(locator='Investigators/Winawer', session=None, subject='ke6hl9lm')
INFO: Need to process 5 study sessions
INFO: PROCESSING STARTS: {'subject': 'dbictest1', 'outdir': '/var/home/gryan/src/github.com/rh-impact/outgoing/head/advanced applications libraries', 'session': None}
INFO: Processing 1 pre-sorted seqinfo entries
INFO: Processing 1 seqinfo entries
INFO: Doing conversion using dcm2niix
INFO: Converting /var/home/gryan/src/github.com/rh-impact/outgoing/head/advanced applications libraries/sub-dbictest1/anat/sub-dbictest1_scout (1 DICOMs) -> /var/home/gryan/src/github.com/rh-impact/outgoing/head/advanced applications libraries/sub-dbictest1/anat . Converter: dcm2niix . Output types: ('dicom',)
INFO: Adding "IntendedFor" to the fieldmaps in /var/home/gryan/src/github.com/rh-impact/outgoing/head/advanced applications libraries/sub-dbictest1.
WARNING: We cannot add the IntendedFor field: no fmap/ in /var/home/gryan/src/github.com/rh-impact/outgoing/head/advanced applications libraries/sub-dbictest1
INFO: Populating template files under /var/home/gryan/src/github.com/rh-impact/outgoing/head/advanced applications libraries
INFO: PROCESSING DONE: {'subject': 'dbictest1', 'outdir': '/var/home/gryan/src/github.com/rh-impact/outgoing/head/advanced applications libraries', 'session': None}
INFO: PROCESSING STARTS: {'subject': 'o6jgilro', 'outdir': '/var/home/gryan/src/github.com/rh-impact/outgoing/Development/Functional', 'session': None}
INFO: Processing 2 pre-sorted seqinfo entries
INFO: Processing 2 seqinfo entries
WARNING: Could not determine the series name by looking at protocol_name, series_description fields
WARNING: Could not figure out where to stick 1 sequences: ['99-Phoenix Document']
INFO: Doing conversion using dcm2niix
INFO: Converting /var/home/gryan/src/github.com/rh-impact/outgoing/Development/Functional/sub-o6jgilro/anat/sub-o6jgilro_32ch-head-coil_scout (1 DICOMs) -> /var/home/gryan/src/github.com/rh-impact/outgoing/Development/Functional/sub-o6jgilro/anat . Converter: dcm2niix . Output types: ('dicom',)
INFO: Adding "IntendedFor" to the fieldmaps in /var/home/gryan/src/github.com/rh-impact/outgoing/Development/Functional/sub-o6jgilro.
WARNING: We cannot add the IntendedFor field: no fmap/ in /var/home/gryan/src/github.com/rh-impact/outgoing/Development/Functional/sub-o6jgilro
INFO: Populating template files under /var/home/gryan/src/github.com/rh-impact/outgoing/Development/Functional
INFO: PROCESSING DONE: {'subject': 'o6jgilro', 'outdir': '/var/home/gryan/src/github.com/rh-impact/outgoing/Development/Functional', 'session': None}
INFO: PROCESSING STARTS: {'subject': 'phantom1sid1', 'outdir': '/var/home/gryan/src/github.com/rh-impact/outgoing/Halchenko/Yarik/950_bids_test4', 'session': 'localizer'}
INFO: Processing 2 pre-sorted seqinfo entries
INFO: Processing 2 seqinfo entries
INFO: Doing conversion using dcm2niix
INFO: Converting /var/home/gryan/src/github.com/rh-impact/outgoing/Halchenko/Yarik/950_bids_test4/sub-phantom1sid1/ses-localizer/anat/sub-phantom1sid1_ses-localizer_scout (1 DICOMs) -> /var/home/gryan/src/github.com/rh-impact/outgoing/Halchenko/Yarik/950_bids_test4/sub-phantom1sid1/ses-localizer/anat . Converter: dcm2niix . Output types: ('dicom',)
INFO: Converting /var/home/gryan/src/github.com/rh-impact/outgoing/Halchenko/Yarik/950_bids_test4/sub-phantom1sid1/ses-localizer/fmap/sub-phantom1sid1_ses-localizer_acq-3mm_phasediff (1 DICOMs) -> /var/home/gryan/src/github.com/rh-impact/outgoing/Halchenko/Yarik/950_bids_test4/sub-phantom1sid1/ses-localizer/fmap . Converter: dcm2niix . Output types: ('nii.gz', 'dicom')
220918-00:31:22,175 nipype.workflow INFO:
	 [Node] Setting-up "convert" in "/tmp/dcm2niixrb4rm1dc/convert".
INFO: [Node] Setting-up "convert" in "/tmp/dcm2niixrb4rm1dc/convert".
220918-00:31:22,185 nipype.workflow INFO:
	 [Node] Executing "convert" <nipype.interfaces.dcm2nii.Dcm2niix>
INFO: [Node] Executing "convert" <nipype.interfaces.dcm2nii.Dcm2niix>
220918-00:31:22,415 nipype.interface INFO:
	 stdout 2022-09-18T00:31:22.415072:Chris Rorden's dcm2niiX version v1.0.20220720  (JP2:OpenJPEG) (JP-LS:CharLS) GCC5.5.0 x86-64 (64-bit Linux)
INFO: stdout 2022-09-18T00:31:22.415072:Chris Rorden's dcm2niiX version v1.0.20220720  (JP2:OpenJPEG) (JP-LS:CharLS) GCC5.5.0 x86-64 (64-bit Linux)
220918-00:31:22,415 nipype.interface INFO:
	 stdout 2022-09-18T00:31:22.415072:Found 1 DICOM file(s)
INFO: stdout 2022-09-18T00:31:22.415072:Found 1 DICOM file(s)
220918-00:31:22,415 nipype.interface INFO:
	 stdout 2022-09-18T00:31:22.415072:Convert 1 DICOM as /var/home/gryan/src/github.com/rh-impact/outgoing/Halchenko/Yarik/950_bids_test4/sub-phantom1sid1/ses-localizer/fmap/sub-phantom1sid1_ses-localizer_acq-3mm_phasediff_heudiconv786_e2_ph (64x64x1x1)
INFO: stdout 2022-09-18T00:31:22.415072:Convert 1 DICOM as /var/home/gryan/src/github.com/rh-impact/outgoing/Halchenko/Yarik/950_bids_test4/sub-phantom1sid1/ses-localizer/fmap/sub-phantom1sid1_ses-localizer_acq-3mm_phasediff_heudiconv786_e2_ph (64x64x1x1)
220918-00:31:22,420 nipype.interface INFO:
	 stdout 2022-09-18T00:31:22.420054:Compress: "/usr/bin/pigz" -n -f -6 "/var/home/gryan/src/github.com/rh-impact/outgoing/Halchenko/Yarik/950_bids_test4/sub-phantom1sid1/ses-localizer/fmap/sub-phantom1sid1_ses-localizer_acq-3mm_phasediff_heudiconv786_e2_ph.nii"
INFO: stdout 2022-09-18T00:31:22.420054:Compress: "/usr/bin/pigz" -n -f -6 "/var/home/gryan/src/github.com/rh-impact/outgoing/Halchenko/Yarik/950_bids_test4/sub-phantom1sid1/ses-localizer/fmap/sub-phantom1sid1_ses-localizer_acq-3mm_phasediff_heudiconv786_e2_ph.nii"
220918-00:31:22,420 nipype.interface INFO:
	 stdout 2022-09-18T00:31:22.420054:Conversion required 0.008733 seconds (0.003917 for core code).
INFO: stdout 2022-09-18T00:31:22.420054:Conversion required 0.008733 seconds (0.003917 for core code).
220918-00:31:22,484 nipype.workflow INFO:
	 [Node] Finished "convert", elapsed time 0.048871s.
INFO: [Node] Finished "convert", elapsed time 0.048871s.
Traceback (most recent call last):
  File "/usr/lib64/python3.10/shutil.py", line 815, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/tmp/dcm2niixrb4rm1dc/convert/provenance.ttl' -> '/var/home/gryan/src/github.com/rh-impact/outgoing/Halchenko/Yarik/950_bids_test4/sub-phantom1sid1/ses-localizer/fmap/sub-phantom1sid1_ses-localizer_acq-3mm_phasediff_prov.ttl'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/home/gryan/.python-virtualenv/bin/heudiconv", line 8, in <module>
    sys.exit(main())
  File "/var/home/gryan/.python-virtualenv/lib/python3.10/site-packages/heudiconv/cli/run.py", line 24, in main
    workflow(**kwargs)
  File "/var/home/gryan/.python-virtualenv/lib/python3.10/site-packages/heudiconv/main.py", line 379, in workflow
    prep_conversion(sid,
  File "/var/home/gryan/.python-virtualenv/lib/python3.10/site-packages/heudiconv/convert.py", line 207, in prep_conversion
    convert(cinfo,
  File "/var/home/gryan/.python-virtualenv/lib/python3.10/site-packages/heudiconv/convert.py", line 531, in convert
    res, prov_file = nipype_convert(item_dicoms, prefix, with_prov,
  File "/var/home/gryan/.python-virtualenv/lib/python3.10/site-packages/heudiconv/convert.py", line 716, in nipype_convert
    safe_movefile(op.join(convertnode.base_dir,
  File "/var/home/gryan/.python-virtualenv/lib/python3.10/site-packages/heudiconv/utils.py", line 413, in safe_movefile
    return _safe_op_file(src, dest, "move", overwrite=overwrite)
  File "/var/home/gryan/.python-virtualenv/lib/python3.10/site-packages/heudiconv/utils.py", line 435, in _safe_op_file
    getattr(shutil, operation)(src, dest)
  File "/usr/lib64/python3.10/shutil.py", line 835, in move
    copy_function(src, real_dst)
  File "/usr/lib64/python3.10/shutil.py", line 434, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib64/python3.10/shutil.py", line 254, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/dcm2niixrb4rm1dc/convert/provenance.ttl'

Platform details:

Choose one:

• Local environment

OS version: Fedora 36
Python version: 3.10.6

• Container

• Heudiconv version:

0.11.3

[alexpron]

Dear Heudiconv developers,
I am actually experiencing the same bug with latest (1.1.0) heudiconv package, the provenance file can not be found due invalid cross device link.
Best,