Skip to content

Copy the input into the subject directory instead of re-encoding it - #875

Open
m-reuter wants to merge 8 commits into
Deep-MI:devfrom
m-reuter:fix001
Open

Copy the input into the subject directory instead of re-encoding it#875
m-reuter wants to merge 8 commits into
Deep-MI:devfrom
m-reuter:fix001

Conversation

@m-reuter

@m-reuter m-reuter commented Sep 7, 2026

Copy link
Copy Markdown
Member

mri/orig/001.mgz is meant to be a copy of the input, but it was a read-and-write. For a scaled
NIfTI, whose array nibabel hands back as float64, MGH could not store the requested type and the
write raised inside a discarded Future, so the file was missing for every .nii.gz input with
no error anywhere. #874 stopped the crash, which only turned the missing file into a 28.8 MB float32
stand-in for a 23.8 MB int16 input.

The fix

Two files with two jobs, both written by run_fastsurfer.sh, the only place that knows the input:

file what it is
mri/orig/001.<ext> byte-for-byte copy of the input, in the format it arrived in. Nothing reads it back.
mri/rawavg.mgz the copy the tools read. Symlink to the archive when the input is .mgz, converted otherwise.

A copy cannot lose anything and needs no data-type policy. rawavg.mgz has to keep exactly that
name because pctsurfcon builds the path itself, so a rawavg.nii.gz would silently break
gray/white contrast. That is what makes the two files distinct rather than redundant.

Verified on both real quicktest inputs: 001 byte-identical in each case, rawavg.mgz converted to
>f4 with fov=256.0 for the 0.8mm scaled NIfTI and a symlink for the 1.0mm MGH.

Why the conversion is ours

The segmentation block has no FreeSurfer dependency, so mri_convert is out, and both Python
candidates drop the field of view that #873 fixed in our writer:

writer dtype fov
nib-convert >f4 0.0
neuroreg.image.save_image 0.6.2 >f4 0.0
save_image (ours) >f4 256.0

Voxel values are identical in all three. Switching to neuroreg is a one-line change once the fix on
its main branch ships. This also fixes fov=0 on the T2 copy, which went through nib-convert.
recon-all converts its own -T2 input with --no_scale 1, so keeping the input's values rather
than rescaling them is also what FreeSurfer does at this step.

Also in this PR

  • The silent failure itself. The conformed orig.mgz write had the same discarded Future. It is
    now drained by main with the others, so a failed write reaches the exit code. That mechanism is
    why this bug survived two releases.
  • T2 gets the same treatment, at the names recon-all uses: mri/orig/T2raw.<ext> for the copy
    and mri/orig/T2raw.mgz for the converted one, which N4 reads. recon-all converts a -T2 input
    to that path with --no_scale 1, and samseg and -T2pial look for it there. For an .mgz T2
    the two collapse into one file. HypVINN is unaffected: it reads the bias-corrected
    mri/T2_nu.mgz, not the raw copy.
  • copy_orig_name removed from run_prediction.py and common.py, about 35 lines. Two
    mechanisms writing one path would otherwise race. run_prediction.py standalone no longer writes
    001, which is right for something that is not a pipeline.
  • recon-surf.sh keeps the rawavg it is given rather than linking orig.mgz over it, and warns
    and falls back to the old behaviour when there is none, so --surf_only on a directory prepared
    elsewhere does not regress.
  • CerebNet/config/dataset.py and EDITING.md point at mri/rawavg.mgz, since 001 no
    longer has a fixed name.

Impact

New per subject: mri/rawavg.mgz for --seg_only runs. Renamed: mri/orig/001.mgz to
001.<ext>, and mri/orig/T2.001.mgz to mri/orig/T2raw.<ext> plus mri/orig/T2raw.mgz. The T2
names were FastSurfer's own invention and now match recon-all.

rawavg.mgz now holds the input rather than the conformed image, so ?h.w-g.pct is sampled from
the input. The contrast is scale invariant, so this is quantisation noise rather than a shift:
conform's rescale is a pure multiplication, since src_min measures 0 on both real inputs, and
100*(w-g)/(0.5*(w+g)) cancels it exactly. For the 1.0mm case the input is already conformed uchar,
so nothing changes. Neither ?h.w-g.pct.stats has a quicktest tolerance yaml, so adding that
coverage is separate work and not in this PR.

The quicktest reference will differ: mri/orig/001.* reappears for the 0.8mm case and rawavg.mgz
becomes a real file rather than a link.

Tests

test/image/test_copy_input.py, 9 cases: the copy is byte-identical for a scaled NIfTI and for an
MGH input, compound extensions survive so .nii.gz does not become .gz, the converted rawavg
carries the right dtype and fov and preserves values, the MGH rawavg is a relative symlink so
the subject directory stays movable, re-running swaps a symlink for a real file and back, a NIfTI T2
produces both the archive and the conversion while an MGH T2 needs only one file, and a missing
input is reported before anything is written so a typo leaves no half-built directory.

expected-files.yaml gains a file_patterns section, since an exact-set check cannot express a name
that depends on the input, and test_file_existence checks those globs.

543 tests pass across test/image, test/config and test/shell/test_bash4_lint.py. Ruff and
codespell clean, both shell scripts pass bash -n. The wiring in run_fastsurfer.sh and
recon-surf.sh has no unit coverage, so a quicktest run is the first real check on it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant