Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

General discussion about qform/sform matrices and codes #3283

Open
5 of 12 tasks
jcohenadad opened this issue Mar 14, 2021 · 5 comments
Open
5 of 12 tasks

General discussion about qform/sform matrices and codes #3283

jcohenadad opened this issue Mar 14, 2021 · 5 comments

Comments

@jcohenadad
Copy link
Member

jcohenadad commented Mar 14, 2021

Description

There has been a lot of issues related to the NIfTI's qform/sform and qform_code header fields. The purpose of this issue (which is not really an issue) is to centralize all the relevant information and cross-reference related issues. This issue also gathers various strategies for how to best deal with qform/sform fields.

Here are relevant links:

Resolved issues:

Open issues:

NB: Issue #4132 is simply a combination of #3005 and #4134, and so isn't listed above.

What do other neuroimaging software do?

nibabel (used by SCT):
The algorithm to deal with qform/sform is defined in the get_best_affine() method. It is:

  1. If sform_code != 0 (‘unknown’) use the sform affine; else
  2. If qform_code != 0 (‘unknown’) use the qform affine; else
  3. Use the fall-back affine.
    Which means that sform gets precedent on the qform.

ITK (used by ANTs):

  • uses qform whenever qform_code > 0.
  • The sform will be overwritten in output images, and sform code will be set to 0. If the sform is not identical to the qform, information will be lost.

MRtrix:

  • When writing NIFTI files: write the same transformation to both the qform and sform fields, based on MRtrix3’s internal representation of the header transformation for that particular image.
  • Set both sform_code and qform_code to 1, indicating that they both provide transformations to scanner-based anatomical coordinates.
    This provides the least ambiguity for image data generated by MRtrix3, and ensures maximal compatibility with other software tools (which may only attempt to read from one field or the other).

fMRIprep:
TODO

Possible solutions

@joshuacwnewton
Copy link
Member

joshuacwnewton commented Mar 14, 2021

I've been thinking about this a lot recently, too, see #3270 (comment).

As a bare minimum, I was thinking that we could come up with a set of checks to see whether the header complies with NIFTI specifications. Then, apply these checks when saving/loading images to catch issues. And, perhaps emit only DEBUG messages for now to keep the noise to a minimum.

But, checks like these would just be for getting an idea of the current state of SCT. (What problems exist, which parts of SCT are affected, etc.) I'm not sure what to do about those problems yet. I'm glad you've shared approaches from other software.

@joshuacwnewton
Copy link
Member

joshuacwnewton commented Mar 24, 2021

One of the qform/sform issues linked above (#3232) sparked a discussion thread over at nibabel: nipy/nibabel#1001

That issue contains some (admittedly frank) comments that nonetheless I appreciate hearing.

The problem boils down to a combination of two coding patterns that are in common use:

  • Using get_fdata to retrieve the image data, so that it is always loaded and manipulated as float64, regardless of the storage type.

  • When creating a new image from an existing image (e.g. after some processing step), using the header from the original image to preserve the NIFTI header metadata (not that there's much of use in there, but that's another matter):

That last comment piqued my interest. Are SCT's current practices frowned upon? I'm curious how the nibabel developers expect the nibabel.Nifti1Image class to be handled vs. how we're actually handling it.

I'm also wondering... If we were using nibabel.Nifti1Image directly instead of our own custom Image class, and using their battle-tested methods for headers/affines, would we encounter fewer header issues?

Some extra benefits to this:

  • We would more closely adhere to nibabel's documentation, which IMO is excellently written and provides very helpful guidance for grad students/devs/etc. who may be new to medical imaging data formats.
  • It would nudge us to get more involved in the nibabel project (opening issues we encounter, discussing typical usage patterns). Given that we already depend heavily on the project, this seems like a net benefit for everyone involved.
  • We would avoid other Image-related gotchas such as Image class pre-loads data when not needed, slowing down sct_apply_transfo (and others) #3271 (comment).

(Switching from Image to Nifti1Image in our code would be quite a large undertaking, though. It certainly sounds nice, but... priorities and resources, etc.)

@jcohenadad
Copy link
Member Author

i have nothing more to say on #3283 (comment), i agree 100% with your comment @joshuacwnewton

@joshuacwnewton
Copy link
Member

joshuacwnewton commented Mar 25, 2021

I'm curious how the nibabel developers expect the nibabel.Nifti1Image class to be handled vs. how we're actually handling it.

On this note, while exploring the NiPreps ecosystem, I came across the niworkflows project. There is overlap between niworkflows and core nibabel developers, so it seems like a good resource to refer to for imaging workflows involving nibabel.

Notably, It has an interfaces/header.py module containing some interesting qform/sform handling code. For example, it has classes called ValidateImage and SanitizeImage that check for xform correctness, and look very appropriate for the issues we've been having. (Interesting and relevant discussion: nipreps/fmriprep#873 (comment))

Even for something as simple as "copying an affine matrix from a reference image" they do things a little differently than we do. Compare our current approach (from #2400):

# Copy q/sform and code
self.hdr.set_qform(im_ref.hdr.get_qform())
self.hdr._structarr['qform_code'] = im_ref.hdr._structarr['qform_code']
self.hdr.set_sform(im_ref.hdr.get_sform())
self.hdr._structarr['sform_code'] = im_ref.hdr._structarr['sform_code']

With their internal function _copyxform:

    # Copy xform infos
    qform, qform_code = orig.header.get_qform(coded=True)
    sform, sform_code = orig.header.get_sform(coded=True)
    header = resampled.header.copy()
    header.set_qform(qform, int(qform_code))
    header.set_sform(sform, int(sform_code))
    header["descrip"] = "xform matrices modified by %s." % (message or "(unknown)")

Little things like the usage of the setters/getters, as well as updating the header description, seem like valuable takeaways. All in all, I'm really interested in exploring NiPreps further.

@joshuacwnewton
Copy link
Member

joshuacwnewton commented Mar 28, 2021

Notably, It has an interfaces/header.py module containing some interesting qform/sform handling code. For example, it has classes called ValidateImage and SanitizeImage that check for xform correctness, and look very appropriate for the issues we've been having. (Interesting and relevant discussion: nipreps/fmriprep#873 (comment))

Small update: I expected to be able to call ValidateImage on an image, but quickly learned that utilities from niworkflows are intended to be used as part of nipype-style pipelines.

Description of nipype for the curious

Quoting from the link above

So niworkflows is a couple things, including a staging ground for interfaces and small workflows that we might want to merge into nipype one day. But most importantly, it’s what we need to make fmriprep and mriqc work, which includes updates to nipype on a much shorter time scale than nipype’s release cycle. We certainly intend everything in it to be open and reusable, but it should be understood to depend on features that are not in a released version of nipype, and we’ve intentionally sacrificed usability of niworkflows as a library for the smooth installation of fmriprep and mriqc.

So, nipype is a framework with its own set of concepts including "interfaces", "nodes", and "workflows". Reading the documentation, I get the impression that these are heavy-duty tools designed to be mapped across large-scale datasets, rather than individual images.


Because of this, connecting niworkflows with SCT isn't as feasible as I'd first hoped. Still, I think their source code will be a useful resource to read and learn from.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants