Adding Images.jl support

[Tokazama]

I'm trying to update this package and add Images.jl support. I've gotten to the point where I can get include("./src/NIfTI.jl") working on Julia 1.0. However, I'm not entirely sure how to proceed with incorporating Image.jl support.

It's suggested here that image types have only two fields (properties and data). I figured I could proceed with one of the following strategies

1. The current header and extensions field could be combined into the properties field so that NIVolume <: AbstractImage.
2. Have some function that maps the current structure to a proper image type?
3. Different load options for including all of the header information (current) and another option that reads to and AbstractImage subtype.

Thanks in advance

[tknopp]

Ping @timholy on this subject.

My point of view is the following: I would like it if we could read a nifti file and the result will be an ImageMetadata object. several thing like image dimension and so on will be encoded into the AxisArray that the ImageMetadata wraps.

One issue I am unclear about is how the metadata should be presented. Will the resulting image object look similar if we load a Nifti or a DICOM file? Do we perform some "normalization" on the parameters?

The end goal from my perspective should be

im = load("file.ni")
save("file.dcm", im)

There can still be something like NIVolume <: AbstractImage which would, however, be only necessary for partial reading of files. The ImageMetadata would then be kind of a materialization.

[timholy]

@Tokazama, oh dear, sorry, those docs are ancient. I didn't even realize they were still part of the repository. The Images.jl homepage does link to https://juliaimages.github.io/latest/ which is current, but we should delete those docs.

As @tknopp says, using an ImageMetadata (which is basically the same thing as what you were referencing) could fix the issue. Normalization will be important if you want multiple file formats to recognize the format. The only format that returns consistent ImageMetadata now is NRRD, so perhaps check that for hints.

[Tokazama]

I've been going back and forth in my head a lot on this lately and I think the thing to do would be something like a NiftiImage

mutable struct NiftiImage
    data::AxisArray  # contains spacing, timing, orientation
   properties::Dict{String,Any}   # contains Nifti1Extensions, descrip field, intent_* fields, etc

I don't think we should even worry about tracking those fields that are unused in the NIfTI1 standard (eg, glmax, glmin, etc) since when writing the image those will probably just be given default values. So some save function would just check the properties for appropriate fields and extract sform stuff from the AxisArray.

I imagine implementing something similar for DICOMs would result in essentially identical structure for the data field. The properties field would be pretty variable depending on what people continue to add to the DICOM standard.

[timholy]

Assuming there's lots of Nifti/DICOM-specific stuff in the header, I think that's a really good plan. If one wants to then write it out as a DICOM one could use @requires and dispatch on the type.

[tknopp]

Do we really require a dedicated type? Of course dicom and nifti will have additional metadata but having everything resulting in an ImageMetadata object will make the handling much easier from the user perspective. We should introduce required parameters that are normalized when loading and denormalized when saving. This concept for nifti, dicom, nrrd, vtk, and further formats would be awesome.

[timholy]

Hmm, you're right, perhaps run-time checking is a better idea: there's nothing performance-sensitive about testing whether a given image is from a particular file type (you're not checking that on a pixel-by-pixel basis). What about the following standard for the ImageMetadata properties:

Dict{String,Any} with 2 entries:
  "fileformat" => "Nifti"
  "fileheader" => Dict{String,Any}("niftifield1"=>'a',"niftifield2"=>2)

Of course you could extract additional info into other named properties, and perhaps some of these could become semi-standard.

For biomedical image formats, I highly recommend leveraging AxisArrays. If you've not seen it, try

julia> using TestImages

julia> img = testimage("mri");

julia> print(summary(img))
3-dimensional AxisArray{ColorTypes.Gray{FixedPointNumbers.Normed{UInt8,8}},3,...} with axes:
    :P, 0:1:225
    :R, 0:1:185
    :S, 0:5:130
And data, a 226×186×27 Array{Gray{N0f8},3} with eltype ColorTypes.Gray{FixedPointNumbers.Normed{UInt8,8}}

This image is represented in PRS coordinates, hence the naming of the axes. And note the voxel separations encoded in the ranges.

[tknopp]

ping @hofmannmartin who is working together with me on this concept for a different format (MDF). I may also do this for the ISMRMRD format.

[Tokazama]

I don’t think it would really require a dedicated type.

I do think there should be a more modular framework that allows reading just header data. It's not uncommon to want some info from the header before importing the entire image array.

It would also make it easier for other packages to build on NIfTI.jl. I imagine that some would like GIfTI, CIfTI, etc support. This is essentially NIfTI with some other stuff in the extension.

[timholy]

NRRD supports version, header, keyvals, comments = parse_header(io). This pattern works if you know you're parsing a *.nrrd file. However, figuring out the file type is generically supported by FileIO's query, so it really just comes down to deciding what should happen subsequently. If we want something generic then we have to think about what kind of common information could be extracted.

[tknopp]

Maybe an "ImageFormats.jl" package is in order. Either this could bundle everything together ( inclusive testing conversions) or it could be the basis.

[tknopp]

What about the following standard for the ImageMetadata properties:

Dict{String,Any} with 2 entries:
"fileformat" => "Nifti"
"fileheader" => Dict{String,Any}("niftifield1"=>'a',"niftifield2"=>2)

I like this. We have three categories:

• First the things that can be encoded into the AxisArray.
• Then those parameter were we have some common name. For instance I am storing a rotation matrix in the image. We should have a normalized form here. These parameters could be "toplevel" in the dict. I think we can also agree on things like "subject", "study", and things like this.
• Finally, all the format specific things that go into the "fileheader" Dict.

[timholy]

For instance I am storing a rotation matrix in the image. We should have a normalized form here.

It already exists:

help?> spacedirections
search: spacedirections                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                               
  spacedirections(img) -> (axis1, axis2, ...)                                                                                                                                                                                                                                  
                                                                                                                                                                                                                                                                               
  Return a tuple-of-tuples, each axis[i] representing the displacement vector between adjacent pixels along spatial axis i of the image array, relative to some external coordinate system ("physical coordinates").                                                           
                                                                                                                                                                                                                                                                               
  By default this is computed from pixelspacing, but you can set this manually using ImagesMeta.                                                                                                                                                                               
                                                                                                                                                                                                                                                                               
  ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────  
                                                                                                                                                                                                                                                                               
  spacedirections(img)                                                                                                                                                                                                                                                         
                                                                                                                                                                                                                                                                               
  Using ImageMetadata, you can set this property manually. For example, you could indicate that a photograph was taken with the camera tilted 30-degree relative to vertical using

  img["spacedirections"] = ((0.866025,-0.5),(0.5,0.866025))

  If not specified, it will be computed from pixelspacing(img), placing the spacing along the "diagonal". If desired, you can set this property in terms of physical units, and each axis can have distinct units.

[Tokazama]

It looks like a lot of the structure for a standardized format is in place, but I'm not entirely sure of the right way to go about conforming to this. This has been my thought process:

1. I want to be able to read an image into a format that is compatible with the entire "Image" library. However, I don't want to require the entire library in order to read my image into a compatible format (e.g. I shouldn't need "ImageFiltering" to read in my image but I want it to be compatible with its functions).
2. So I want my FileIO integrated package to depend on some standard that will predictably work with the beautiful "Image" library. I have "ImageMetadata" to provide the format but need "ImageCore" to provide the standard API. I'm not really sure where "ImageAxis" fits into this yet because my understanding of the documentation makes it seem really similar to "AxisArrays".
3. I make ImageMeta the final import type and write from ImageMeta type with the appropriate "fileformat" property. I have some checks for other appropriate properties in the writing step.

I'm sure most of what's necessary to move forward is in place and additional clarity will arise as more unique imaging formats are implemented. However, it's not entirely clear what is necessary to be Image library "compliant" or if that's even supposed to be a thing.

[timholy]

ImageCore = conversions/reinterpretations of values + frequently-used traits
ImageAxis = AxisArrays + trait specializations for AxisArrays
ImageMetadata = ImageAxis + addition info, including additional trait specializations

You shouldn't need anything higher in the stack than this. Here are ImageMetadata's requirements. No ImageFiltering/Images/etc.

[tknopp]

@Tokazama: Tim has designed Images "version 2" two years ago exactly in such a way that the dependencies are minimalistic. Don't get blended from the number of packages involved, this is more a sign of good modularity. Actually this is why I want to push the discussion into making "ImageMetadata" standard format for storing complex files like nifti and dicom.

I don't have the bandwidth to contribute much code right now but it would be absolutely awesome if you could play with the idea for NiFTi.jl. I can give you rights such that you can work on a branch.

For the record, here (https://github.com/MagneticParticleImaging/MPIFiles.jl/blob/master/src/Image.jl) you can find some code where I have the correspondence (ImageMetadata <-> MDF File). MDFs is a domain specific image format that I have developed.

[Tokazama]

I'd be happy to put a bit of time into that.

[tknopp]

@Tokazama you should now have rights to make changes to this package. Best thing for new ideas is creating a branch and working there.

[Tokazama]

tldr: I'm working on this again. It's not ready yet but it should be ready for heavy use very soon.

Apologies for the long wait on this. Classes and projects put this on hold for a while. Here are some of the major changes in the most recent update and what the reasoning behind them were.

1. Generic backend for image reading (ImageFormats): Once all the header information is in place the ImageStream hands off the process to ImageFormats. I'm not sure whether this ends up being useful for other packages yet, but the implementation seems basic enough to use in other code bases without much effort.
2. Dropped theNiftiHeader types: The use of ImageFormats means that once all the info is available in an ImageStream there is no use for a NIftiHeader. It would only serve as an intermediate container. The one downside to this is that it may make readability of code a bit harder. I'm trying to make this as clear as possible in the code itself so others can tell what the basic outline of the header is still.
3. Writing files is not yet resolved so use at your own caution. Hopefully I'll figure this out today or tomorrow.
4. Transforms and sinks: I decided on going forward with ImageFormats because the NIfTI standard has a stupid amount of data types it tries to support. I've been doing some research on it and have found out that many of them are in fact used but sometimes by one or two programs out in the wild. This is unfortunately the nature of the beast. Something like load(file, sink::AbstractArray) becomes very attractive in this situation. Unfortunately I only have limited support for this now. I'm going to be looking at NamedDims.jl for simplifying movement between bits types and structs (like RGB, RGBA, Quaternion, Triangle, Point, tons of stats distributions,...). This will just take the dimension names (or axisnames if you prefer) and permutedims and convert to the proper element type (like channelview and colorview)

I still need to put together proper tests and fix a lot of the documentation, but I should actually be able to work on this again now that I have a bit more time.

[tknopp]

Great progress. Isn't NamesDims.jl just an alternative to AxisArrays.jl? The later also encodes the dim name with zero cost abstraction.

I am currently using NIfTI.jl here
https://github.com/MagneticResonanceImaging/MRIReco.jl/blob/master/src/IO/Nifti.jl#L9
in my MRI reconstruction package. There I need to create a NIfTI file "from scratch" although it possible to retrieve some additional metadata from the underlying ISMRMRD raw data file. Once you have proper write support, I am looking forward learning the interface how to write the NIFTI file.

[Tokazama]

NamedDims.jl is supposed to be the future implementation of NamesDims.jl and have zero cost at compile time (see JuliaCollections/AxisArraysFuture#1 (comment)). I'm not sure if this will happen anytime soon, but the functionality I'd like to get out of it isn't necessary for most use cases, so that's a minor issue for now.

The goal is to have a generic interface for any AbstractArray so that niwrite(::String, ::AbstractArray) just works. The orientation and axes are acquired using the ImageMeta interface. Looking at your link you could probably just wrap niwrite in saveImage.

[tknopp]

Ok, I missed that new direction for AxisArrays. Whatever is used in the end. Most important is that we agree on a single standard, so that this remains compatible throughout different packages.

Of what type should the AbstractArray be if I want to provide some control over how the data is stored? should / can ist be an ImageMeta that wraps an AxisArray? Will this store the pixelspacing and image center correctly? Then I would be interested how to report the orientation in the ImageMeta.

[Tokazama]

Yeah, ImageMeta{T,N,<:AxisArray} is probably the way to go. I'm trying to use the same syntax that is used in Images. I figure that no matter how AxisArrays develops in the future that working around these standard functions will be easiest. The orientation is acquired by calling spacedirectionsfrom ImageCore.

[Tokazama]

I figured there needed to be an update on this as my progress has been embarrassingly slow. Fortunately, I mostly just have to write a couple papers and work on this over the next two weeks. I actually was using my branch pretty heavily at one point (feeding thousands of images into a Flux model), so I don't expect there to be any major issues for people trying to do their day to day stuff. I've also enlisted a physics guy to stress test it with some multi-gigabyte DTI data he's working on. Hopefully, this will cover any major blind spots I may have.

I'm going go over my plan moving forward on this as it appears some people may be waiting for me to actually get into gear and do what I actually said I'd do with this :P. I've ordered it according to priority so others may have an idea of when to expect things.

1. Basic I/O support using FilIO syntax
   • NIfTI-1 support
   • NIfTI-2 support
   • Prepare for multi-threading: there are some points in the code that need to be refactored so multi-threading is easier to implement for the approaching 1.3 release.
   • Implement Brain imaging data structure support (likely in a different package that this could depend on)
     • It turns out file extensions are very important for NIfTI. BIDS seems to be the most sensible approach to doing this because it relies heavily on file paths and extensions to determine organization. Therefore, it can assist in identifying what format a file should be loaded as before parsing the entire header.
     • This will also help standardize the metadata that shoud be kept around and that needs some sort of designated syntax to access.
   • Documentation
     • Method documentation: There's a lot of documentation I have floating around on this but it's a mess and needs to be supplemented.
     • Examples: It would be nice to have some examples where we demonstrate how to straightforward it is to load an image and do some traditional processing. I was working on deriving the various tensors from a DWI image at one point but I may just have to do a simulated mass univariate analysis.
2. Support extensions to the NIfTI specification
   • CIfTI: Used commonly in the Human Connectome Project which I'm currently working with and need a way to get my data into Julia.
   • GIfTI: Very similar implementation to CIfTI but with some slight differences, so work on these two should progress together closely.
3. Visualizations/plotting
   • There are currently methods that aren't too difficult for perusing images by slices. However, It would be very useful to have mesh plots for looking at regions of interest. Some of these CIFTI/GIFTI types also use point clouds and triangle surfaces.

I expect to finish the first one fairly soon (weeks). I have no time table for the second or third one but I need them for several other projects that are coming up this next month. So if they don't get done soon that there will be some alternative strategy that fulfills a similar role.

[tknopp]

Hey, @Tokazama. Maybe it makes sense to chunk this a bit. I think 1. + some minimal form of documentation would be most important since it's the interface that people will rely on. Multithreading, 2. and 3. could perfectly follow afterwards.

If you are fine with the interface, tests are passing just go ahead merge and release.

[Tokazama]

I agree. I want to get the first out the door ASAP. I'm putting together the BIDS stuff here for now. Right now I'm just outlining everything so that it doesn't have to be rewritten every time a new format is supported. I don't plan to actual finish any more than the MRI relevant specification before getting the NIfTI branch ready to go.

[timholy]

It turns out file extensions are very important for NIfTI

I am sure you have seen this. If it helps, you could add a detection function to FileIO and then use its query interface.

[Tokazama]

Yes I have. And that's useful for the vast majority of cases, but there's a whole slew of CIfTI formats that require "somename.unique_variant.nii" so it requires an additional parsing of the "unique_variant" portion in addition to detecting the "nii". This also tells us it's likely a NIfTI-2 format because CIfTI is exclusively NIfTI-2 format. There are also unique combinations where if you have a certain set of extension then you expect to have another extension type file in the same folder (kind of like how Analyze requires a ".hdr" and ".img").

It's not something that will hold me up on getting the first release out the door too much, but I need to at least have a logical way to change how header data is handled based on the file extension. Especially the "extension" of the NIfTI file between the raw data and header. That stuff get's a little crazy.

[Tokazama]

BTW. My latest stuff is nearly ready to be merged to my branch and once those tests are passing I'll merge. The only hold up at this point is getting NamedDims, AxisArrays, etc. compatibility finalized.

[timholy]

The only hold up at this point is getting NamedDims, AxisArrays, etc. compatibility finalized.

So we can look forward to merging in 2021? 😄 I should engage a bit more with that process, it's clearly important. So little time though...

[Tokazama]

I'm finishing up a pretty thorough write up on how to move that ahead that I'll post to relevant channels (hopefully today). I have to finish a fairly important analysis today, but once that's finished I'll push stuff to my branch. It probably won't work outside of my local packages without a the array stuff in place, but at least people could take a look at the method names and syntax so that we're sure they gel right with everyone.

[tknopp]

Hey @tanmaykm,

One of the prime reason I am looking forward to your work being merged is that I would like to get rid of a custom Nifti writer that I have once written. I have moved the (ugly) code to a public location:

https://github.com/tknopp/ImageUtils.jl/blob/master/src/AnalyzeNifti.jl

I now have some questions. My Type is an ImageMeta{...AxisArray}. The ImageMeta contains the extra stuff, like rotation matrices. The AxisArray is a 4 or 5-dim type:

https://github.com/tknopp/ImageUtils.jl/blob/master/src/Arrays.jl#L10

The dimensions might have a unit or not. So now the question is: How can I write such a type to a nifti using your new API?

Next question: My export function also allows to write colorized RGB{N0f8} arrays. Do you already support this?

Even if your branch is not feature complete, it would be great if we could merge this and then move on from there.

[Tokazama]

I'll push something to my branch by the end of the night. It's likely not going to work because I need to get some other packages working with it first, but it should be a good place to start. In other words most of what I have for NIfTI.jl is ready but its dependencies aren't.

1. The unit of the dimension appears to be spatial or time right? This will be/is supported through Unitful.
2. I'm using ImageMeta for properties but I'm using the HasProperties trait in ImageCore so if you want to extract this from a different object with the same field then that will be possible.
3. I do support writing color but the NIfTI format limits this to Float32 so that would have to be the storage format.

The biggest changes you'll see overall are that I'm trying to assign a dedicated trait method (similar to spacedirections) for all the properties we care about and some more options for loading and saving images. In so doing I'm also enforcing what Type may be written and received through that method. This gets around type stability for the dictionaries that story Any values.

Also, thank you for those links. One of the biggest problems is that I think there should be a more established set of properties among MRI images if we want Julia and MR to really take off. The problem is that I've spent most of my time working on the image analysis side of things and I only know from collaboration and preprocessing pipelines what goes into the image acquisition. It's entirely possible that I miss some very useful ways of interacting with images.

[tknopp]

The unit of the dimension appears to be spatial or time right? This will be/is supported through Unitful.

yes, it is space and time. Its important to get this into a canonical form before writing. If an axis has no unit but is spatial (x,y,z) then it would be great if you would consider this to be meters, time should be seconds.

I'm using ImageMeta for properties but I'm using the HasProperties trait in ImageCore so if you want to extract this from a different object with the same field then that will be possible.

That sounds great. But I a wondering: what will the object look like when I am reading a NIfTI file? Will it be a nifty specific type? If yes, will there be a conversion into some generic type (ImageMeta) possible?

I do support writing color but the NIfTI format limits this to Float32 so that would have to be the storage format.

There was #define DT_RGB24 128 which seems to work. Note that this has a memory representation of 24 bits, while sometimes (I think in Cairo) this has 32 bit. ImageJ is able to read it. I am reading such files here:
https://github.com/tknopp/ImageUtils.jl/blob/master/src/AnalyzeNifti.jl#L175

The biggest changes you'll see overall are that I'm trying to assign a dedicated trait method (similar to spacedirections) for all the properties we care about and some more options for loading and saving images. In so doing I'm also enforcing what Type may be written and received through that method. This gets around type stability for the dictionaries that story Any values.

While I think a proper trait system is nice, my impression is that we at some point really want a file-independent type which you convert to. if you do image processing you will load the DICOM / Nifti file and will then want to forward that to you image pipeline. It makes me somewhat nervous when I never know what the concrete type is.

Also, thank you for those links. One of the biggest problems is that I think there should be a more established set of properties among MRI images if we want Julia and MR to really take off. The problem is that I've spent most of my time working on the image analysis side of things and I only know from collaboration and preprocessing pipelines what goes into the image acquisition. It's entirely possible that I miss some very useful ways of interacting with images.

I am coming from the other side, I am doing image reconstruction and want to store things in the proper file format. Since you have some MR background you might be familiar with the ISMRMRD file format. In my package "MRIReco.jl" I have implemented full support for but still need something in which to store the reconstruction result. Wouldn't it be great if we could just use "DICOM.jl" and "NiFTI.jl" for that? If we have reached that state we would have an awesome Julia story here.

[Tokazama]

I apologize for the terrible response time on this. I was hoping to get the array interface resolved but I think there was some loss of momentum in the community on that. I'll just get AxisArrays working for now.

As for concerns about the trait system and knowing exactly what will be returned I have started on this JuliaNeuroscience/NeuroCore.jl#1. It probably won't match perfectly with ISMRMRD and I know there are differences from DICOM (mostly milliseconds to seconds). I've taken a look at your MRIReco.jl package and it differs in breaking properties into individual parameters instead of a dedicated structure for something like acquisition parameters.

[tknopp]

ISMRMRD is a raw data format so its not so relevant here. What I am looking for is an abstraction layer above DICOM / NiFTi. and ideally one has a bi-directional mapping between the files and its Julia representation. Is that what you are aiming for with NeuroCore.jl? Any reason this is specific to Neuro?

[Tokazama]

What I am looking for is an abstraction layer above DICOM / NiFTi.
Yes, the idea is to have family of dedicated methods where the return value is known ahead of time (e.g., type stable).

Any reason this is specific to Neuro?
This is being discussed in the first big PR. There are obviously a lot of components that are not neuro specific that make up the BIDS standard I'm trying to use. I would be happy to have NeuroCore be strictly neuroscience related traits, but then other packages would have to choose to follow the BIDS standard and we can't really enforce that.

We could of course just ignore BIDS as a whole and do our own thing, but the entire point of its creation was that too many groups kept doing that and it made research difficult. If at all possible I'd like to build on the work that the neuroscience community is beginning to adopt instead of reinventing the wheel.

[tknopp]

What is BIDS? Maybe I just did not get the background right here.

[Tokazama]

Brain Imaging Data Standard. I wouldn't pay too much attention to another group trying to push their standard but a lot of software is beginning to incorporate it and a lot of the well funded projects now expect their software to follow it. It doesn't have complete coverage of neuroinformatic modalities but there are quite a few that are actively being worked on here.