with and without --overwrite

[dkp]

I believe you may have dealt with similar topics, so sorry if this is a repetition... I don't see this explicitly in the issues. If I have 2 runs to convert, I expect to get:
sub-220_ses-foo_task-rest_run-01_bold.json
sub-220_ses-foo_task-rest_run-01_bold.nii
sub-220_ses-foo_task-rest_run-01_bold.nii.gz
sub-220_ses-foo_task-rest_run-01_events.tsv
sub-220_ses-foo_task-rest_run-02_bold.json
sub-220_ses-foo_task-rest_run-02_bold.nii
sub-220_ses-foo_task-rest_run-02_bold.nii.gz
sub-220_ses-foo_task-rest_run-02_events.tsv

BUT, if I run without --overwrite, then heudiconv complains that it would have to overwritedata and generates only:
sub-220_ses-foo_task-rest_run-01_bold.json
sub-220_ses-foo_task-rest_run-01_bold.nii
sub-220_ses-foo_task-rest_run-01_bold.nii.gz
sub-220_ses-foo_task-rest_run-01_events.tsv

This is not the behavior I expect. That is, it seems that I have to run with --overwrite or not everything will be converted.
Am I missing something?

[mgxd]

@dkp can you share a few things?

• heudiconv version
• heuristic
• conversion log
• dicominfo

Thanks!

[mgxd]

hi @dkp is this still a problem?

[dkp]

Hi Mathias, Thanks for asking. I check the heudiconv container with docker pull before I run anything, so I was running the latest in December, and it has updated several times between Dec and now. I am still reverse engineering to try to understand what overwrite does. I generally leave it on. I'd love it if you could further explain the behavior of this tag. Most recently, I see that with overwrite, my ASL image writes as two images with one component each. Without overwrite, my ASL image has 2 components and then heudiconv errors out without writing the rest of the images. Here's an example with only 2 dicom datasets: run-01 and run-02, just a few days ago with heudiconv version 0.5.4.dev1 overwrite ON: – sub-219_ses-itbs_acq-asl_run-01.json – sub-219_ses-itbs_acq-asl_run-01.nii.gz – sub-219_ses-itbs_acq-asl_run-02.json – sub-219_ses-itbs_acq-asl_run-02.nii.gz overwrite OFF: – sub-219_ses-itbs_acq-asl_run-01.json – sub-219_ses-itbs_acq-asl_run-01.nii.gz Okay, if overwrite is OFF, then the 2 images are combined into an ASL with 2 components. If overwrite is ON then we get 2 images with one component each AND the post images are written as well. A runtime error is generated: *Runtime**Error**: was asked to convert into /base/Nifti/sub-219/ses-itbs/func/sub-219_ses-itbs_acq-asl.nii.gz but destination already exists* – *make: * [run] **Error** 1* Thanks for your time. Great tool. Here's my documentation on it, BTW: https://neuroimaging-core-docs.readthedocs.io/en/latest/pages/heudiconv.html

…

-Dianne

On Wed, Mar 13, 2019 at 7:49 AM Mathias Goncalves ***@***.***> wrote: hi @dkp <https://github.com/dkp> is this still a problem? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#288 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA66QFLW-iA-8zKn_4ltip55HQKrKnomks5vWQ_XgaJpZM4ZW9gk> .

-- Dianne Patterson, Ph.D. Research Scientist dkp@email.arizona.edu <dkp@u.arizona.edu> or diannepat@gmail.com University of Arizona Speech and Hearing Science 314 1131 E 2nd Street, Building #71 (Just East of Harvill) ============== If you don't hear back from me (and you expected to), I blame the University's new SPAM filter. Please write to my gmail account. ============== Antipiphany: That moment when you realize how little you actually know ==============

[mgxd]

The --overwrite flag is intended to facilitate re-conversions (@yarikoptic feel free to chime in, I know we have pushed and pulled on this before) - without it, heudiconv will not write to file if it already exists. Here is the logic

heudiconv/heudiconv/convert.py

Lines 277 to 304 in 5b7c33e

	if not op.exists(outname) or overwrite:
	tmpdir = tempdirs('dcm2niix')
	# run conversion through nipype
	res, prov_file = nipype_convert(item_dicoms, prefix, with_prov,
	bids, tmpdir, dcmconfig)
	bids_outfiles = save_converted_files(res, item_dicoms, bids,
	outtype, prefix,
	outname_bids,
	overwrite=overwrite)
	# save acquisition time information if it's BIDS
	# at this point we still have acquisition date
	if bids:
	save_scans_key(item, bids_outfiles)
	# Fix up and unify BIDS files
	tuneup_bids_json_files(bids_outfiles)
	if prov_file:
	prov_files.append(prov_file)
	tempdirs.rmtree(tmpdir)
	else:
	raise RuntimeError(
	"was asked to convert into %s but destination already exists"
	% (outname)
	)

What I suspect is happening in your case is 2 (or more) scans are being classified for the same key. This will be caught and rightfully throw an error (stopping any further conversion). This is generally the case when excluding or hardcoding run-<number> in your key. An easy workaround would be to use the {item} formatting key, which is the index of a scan for a given info key. Here's an example

heudiconv/heudiconv/heuristics/bids_ME.py

Line 23 in 5b7c33e

bold = create_key('sub-{subject}/func/sub-{subject}_task-test_run-{item}_bold')

Otherwise, if you could share your heuristic + dicominfo.tsv, we could track down the problem.

And awesome documentation! We were going to write up a readthedocs page, and yours checks off a lot of the boxes! Would you mind if we forked yours as a starting point?

[dkp]

Please, fork away! And thank you for the insights, it may take a while to digest.

…

-Dianne

On Wed, Mar 13, 2019 at 10:30 AM Mathias Goncalves ***@***.***> wrote: The --overwrite flag is intended to facilitate re-conversions ***@***.*** <https://github.com/yarikoptic> feel free to chime in, I know we have pushed and pulled on this before) - heudiconv will not write to file if it already exists. Here is the logic https://github.com/nipy/heudiconv/blob/5b7c33ecfb9f93c547fa5d0a38cbd22e05ea0ebd/heudiconv/convert.py#L277-L304 What I suspect is happening in your case is 2 (or more) scans are being classified for the same key. This will be caught and rightfully throw an error (stopping any further conversion). This is generally the case when excluding or hardcoding run-<number> in your key. An easy workaround would be to use the {item} formatting key, which is the index of a scan for a given info key. Here's an example https://github.com/nipy/heudiconv/blob/5b7c33ecfb9f93c547fa5d0a38cbd22e05ea0ebd/heudiconv/heuristics/bids_ME.py#L23 Otherwise, if you could share your heuristic + dicominfo.tsv, we could track down the problem. And awesome documentation! We were going to write up a readthedocs page, and yours checks off a lot of the boxes! Would you mind if we forked yours as a starting point? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#288 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA66QMUGa5jpR9D89-81mZc23ZSz1t3Cks5vWTL4gaJpZM4ZW9gk> .

-- Dianne Patterson, Ph.D. Research Scientist dkp@email.arizona.edu <dkp@u.arizona.edu> or diannepat@gmail.com University of Arizona Speech and Hearing Science 314 1131 E 2nd Street, Building #71 (Just East of Harvill) ============== If you don't hear back from me (and you expected to), I blame the University's new SPAM filter. Please write to my gmail account. ============== Antipiphany: That moment when you realize how little you actually know ==============