Mean Signal DKI #1230

RafaelNH · 2017-04-22T06:56:49Z

In this PR you can find the implementation of the mean spherical DKI model. This model provides non-Gaussian indexes that do not depend on the orientation distribution fiber and that are less sensitive to artefacts when compared to standard mean kurtosis.

The missing parts to complete this PR are the following:

Add test to cover 100% of the lines of code (97% is the current test coverage)
Documentation proofreading
Example of usage (this should include the comparison between the novel index and the standard mk, and simulations showing the biological relevance of the mean spherical kurtosis model).

Please let me know if you have any additional comments that you want me to address at this point!

Cheers,
Rafael

skoudoro · 2018-04-19T18:44:45Z

Hi @RafaelNH, What is the status of this PR? It would be good if you can complete the missing part :-)
you seem really close to finishing this PR.

nilgoyette

I didn't test anything, it's simply a code review.

nilgoyette · 2018-05-01T15:21:42Z

dipy/reconst/mdki.py

+        if self.min_signal is None:
+            min_signal = MIN_POSITIVE_SIGNAL
+        else:
+            min_signal = self.min_signal


This condition should be set in the constructor, where you write

self.min_signal = self.kwargs.pop('min_signal', None)

Can't you replace None by MIN_POSITIVE_SIGNAL and do a little trick for the ValueError after?

nilgoyette · 2018-05-01T15:25:41Z

dipy/reconst/mdki.py

+        index = index + (slice(None),) * (N - len(index))
+        if model_S0 is not None:
+            model_S0 = model_S0[index[:-1]]
+        return type(self)(self.model, model_params[index], model_S0=model_S0)


I don't see this class subclassed anywhere. If it's true and there's no future plan to subclass it, you should replace type(self) by MeanDiffusionKurtosisFit to avoid useless complexity.

nilgoyette · 2018-05-01T15:27:45Z

dipy/reconst/mdki.py

+        if not mask[v]:
+            continue
+        # Skip if no signal is present
+        if np.mean(msignal[v]) <= min_signal:


Is there no way to zip over mask and msignal at the same time instead of using the v index?

Hi @nilgoyette! I am not finding a trivial way to zip over mask and msignal at the same time! Note that these arrays have different dimensions.

What do you mean, mask is in 3D and msignal in 4D?

Indeed - any concrete suggestion on how to zip over mask and msignal simultaneously?

I never did that in Python. If you say that you searched and found nothing, I'll believe you. Thank you for checking.

nilgoyette · 2018-05-01T15:41:54Z

dipy/reconst/tests/test_mdki.py

+                                              fractions=frac_sph,
+                                              snr=None)
+# Compute ground truth values
+MDgt = f*Di + (1-f)*De


Please stick to the standard that you use everywhere else: spaces between operators. Here and elsewhere.

nilgoyette · 2018-05-01T15:50:00Z

dipy/core/gradients.py

+
+    bmag : int
+        The order of magnitude that the bvalues have to differ to be
+        considered an unique b-value. B-values are also rounded to relative


to relative to

nilgoyette · 2018-05-01T15:54:19Z

dipy/reconst/mdki.py

+
+def mdki_prediction(mdki_params, gtab, S0=1.0):
+    """
+    Predict the mean signal given the parameters of the mean spherical DKI, an


a GradientTable object

nilgoyette · 2018-05-01T15:54:44Z

dipy/reconst/mdki.py

+    Parameters
+    ----------
+    params : ndarray ([X, Y, Z, ...], 2)
+        Array containing in the last axis the mean spherical diffusivity and


nilgoyette · 2018-05-01T15:57:01Z

dipy/reconst/mdki.py

+        return_S0_hat : bool
+            Boolean to return (True) or not (False) the S0 values for the fit.
+
+        args, kwargs : arguments and key-word arguments passed to the


keyword is always written as a single word. If you change it, change it everywhere else in this review.

nilgoyette · 2018-05-01T16:01:58Z

dipy/reconst/mdki.py

+        coefficients of the mean spherical diffusion kurtosis model. Note that
+        nub is the number of unique b-values
+    msignal : ndarray ([X, Y, Z, ..., nub])
+        Mean signal along all gradient direction for each unique b-value


nilgoyette · 2018-05-01T16:04:01Z

dipy/reconst/mdki.py

+        Voxel with mean signal intensities lower than the min positive signal
+        are not processed. Default: 0.0001
+    return_S0_hat : bool
+        Boolean to return (True) or not (False) the S0 values for the fit.


I find this sentence (and others like it) unclear. Why not a simpler "If True, also return S0 values for the fit"?

arokem · 2017-04-23T02:34:11Z

dipy/reconst/mdki.py

+        ----------
+        .. [1] Henriques, R.N., Correia, M.M., 2017. Interpreting age-related
+               changes based on the mean signal diffusion kurtosis. 25th Annual
+               Meeting of the ISMRM; Honolulu. April 22-28


arokem · 2017-04-23T02:34:58Z

dipy/reconst/mdki.py

+            raise ValueError(mes)
+
+    def fit(self, data, mask=None):
+        """ Fit method of the DTI model class


=> "Fit method of the MDKI model class"

arokem · 2017-04-23T02:36:09Z

dipy/reconst/mdki.py

+        # Skip if no signal is present
+        if np.mean(msignal[v]) <= min_signal:
+            continue
+        # Define weights as diag(sqrt(ng) * yn**2)


I think the "sqrt" in this comment is incorrect?

codecov-io · 2019-01-11T02:20:32Z

Codecov Report

❗ No coverage uploaded for pull request base (master@7b76d74). Click here to learn what that means.
The diff coverage is 98.47%.

@@            Coverage Diff            @@
##             master    #1230   +/-   ##
=========================================
  Coverage          ?   83.84%           
=========================================
  Files             ?      120           
  Lines             ?    14561           
  Branches          ?     2294           
=========================================
  Hits              ?    12209           
  Misses            ?     1827           
  Partials          ?      525

Impacted Files	Coverage Δ
dipy/reconst/dki.py	`96.82% <ø> (ø)`
dipy/reconst/fwdti.py	`94.28% <100%> (ø)`
dipy/core/gradients.py	`90% <100%> (ø)`
dipy/reconst/msdki.py	`98.27% <98.27%> (ø)`

RafaelNH · 2019-01-18T16:30:02Z

Hello! Apologies - progress on this was much slower than I've expected. Basically, I've decided to push back on this to double check if recent advances on microstructural modelling were making MSDKI an obsolete and useless technique. However, according to my recent study (Henriques et al., 2019), I can concluded that this type of signal representation approaches are still a valuable tool for the scientific community. Actually, I feel that this is one of the most exciting techniques that I am sharing in an open source environment. Hope you find it interesting as well.

As I've promised you, I've created an example where I've described be relevance of this technique and showed how one can reconstruct diffusion-weighted data using MSDKI. Hope you enjoy.

In my side this PR is ready to go, but let me know if you have further comments.

The tests might not be passing because I am having an intermittent error in dipy.workflows.tests.test_tracking.test_det_track (not related to this PR) in the environment with python 3.5. Should I create a issue on this?

arokem · 2019-01-18T16:34:16Z

Rafael -- nice work! I will take a look later today.

Regarding the error that you hit: did you rebase this on top of master?

RafaelNH · 2019-01-18T16:48:52Z

Hi Ariel! Yes, I did. Is this issue not happening in other PRs?

arokem · 2019-01-18T16:54:50Z

Doesn't seem to be (for example: https://ci.appveyor.com/project/skoudoro/dipy-5dd9b/builds/21721304 from a PR I have in review), but maybe put up an issue about it, and we can try to resolve that together. It might be something intermittent. I agree that it seems unrelated to your changes, so it's a bit puzzling.

RafaelNH · 2019-01-18T17:00:09Z

Well - I did it 9 days ago with no conflict! I've just did it again - still no conflict! Let's see if it passes!

skoudoro

Hi @RafaelNH, Nice work and tutorial! Thank you for this! After a quick look, I added a couple of comments below. I will go deeper later.

Can you add your example in dipy/doc/examples/valid_exemples.txt
Can you add your example on the reconstruction section on dipy/doc/examples_index.rst

skoudoro · 2019-01-18T19:05:39Z

doc/examples/reconst_mdki.py

+import numpy as np
+import matplotlib.pyplot as plt
+
+# Reconstriuction modules


typo: reconstruction instead of Reconstriuction

skoudoro · 2019-01-18T19:07:11Z

doc/examples/reconst_mdki.py

+three different b-values).
+"""
+
+# Sample the spherical cordinates of 60 random diffusion-weighted directions.


typo: coordinates

skoudoro · 2019-01-18T19:21:42Z

doc/examples/reconst_mdki.py

@@ -0,0 +1,350 @@
+"""


Can you add # -*- coding: utf-8 -*- at the top of the file? Otherwise, I get the following error on python27 - windows:

$ python doc/examples/reconst_mdki.py File "doc/examples/reconst_mdki.py", line 324 SyntaxError: Non-ASCII character '\xe2' in file doc/examples/reconst_mdki.py on line 325, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details

skoudoro · 2019-01-18T19:29:03Z

dipy/reconst/tests/test_mdki.py

+from dipy.sims.voxel import multi_tensor_dki
+from dipy.io.gradients import read_bvals_bvecs
+from dipy.core.gradients import (gradient_table, unique_bvals, round_bvals)
+from dipy.data import get_data


get_data is deprecated, can you use get_fnames ?

I do not remember if this deprecation is on master or the last release, I need to check

skoudoro · 2019-01-18T19:34:33Z

dipy/reconst/mdki.py

+
+from dipy.core.gradients import (check_multi_b, unique_bvals, round_bvals)
+from dipy.reconst.base import ReconstModel
+from dipy.reconst.dti import (MIN_POSITIVE_SIGNAL)


I think you do not need the parenthesis

skoudoro · 2019-01-18T19:39:31Z

dipy/reconst/mdki.py

+        args, kwargs : arguments and keyword arguments passed to the
+        fit_method. See mdti.wls_fit_mdki for details
+
+        min_signal : float


This docstring does not appear as __init__ function parameter.

skoudoro · 2019-01-18T20:02:24Z

dipy/reconst/mdki.py

+        self.bmag = bmag
+        self.args = args
+        self.kwargs = kwargs
+        self.min_signal = self.kwargs.pop('min_signal', MIN_POSITIVE_SIGNAL)


ok, I see, I wonder whether it will be better to be explicit and just add this parameter in the __init__ function like you did on line 329

RafaelNH · 2019-01-20T20:09:24Z

Hi @skoudoro! Many thanks for your comments - I've address them all! Let me know if you have further ones!

arokem

Great. I think this is really close to ready to merge. I had just a couple of small comments.

In particular, I wonder whether something could be done to speed up the prediction from this model. Right now, you are looping through voxels (here: https://github.com/nipy/dipy/blob/9a7acfcec8085d0cd9a58156fd1db4e5d7b27a4c/dipy/reconst/msdki.py#L89-L106). Would it be possible to do the dot product over multiple voxels at a time?

At the very least, could we use a mask for this function, to avoid predicting for voxels with no meaningful signal?

arokem · 2019-01-20T21:25:20Z

dipy/reconst/dki.py

@@ -1224,7 +1224,7 @@ def dki_prediction(dki_params, gtab, S0=1.):
        The gradient table for this prediction
    S0 : float or ndarray (optional)
        The non diffusion-weighted signal in every voxel, or across all
-        voxels. Default: 150
+        voxels. Default: 1


Good catch!

arokem · 2019-01-20T21:29:08Z

dipy/reconst/dti.py

@@ -1286,7 +1286,13 @@ def wrapped_fit_tensor(design_matrix, data, return_S0_hat=False,
                                  return_S0_hat=return_S0_hat,
                                  *args, **kwargs)
            data = data.reshape(-1, data.shape[-1])
-            dtiparams = np.empty((size, 12), dtype=np.float64)
+            params0 = fit_tensor(design_matrix, data[0:2],


Is this something to do with reusing this function for other models? Not the standard 12 parameter tensor? Why are you taking two rows from the reshaped data, rather than just the first one?

Indeed, in one of my earliest implementations, I was reusing this function for msdki. I've just realised that I am not using this anymore. Therefore, I've restored dti to the master's branch version.

arokem · 2019-01-21T14:38:11Z

dipy/reconst/msdki.py

+    index = ndindex(msdki_params.shape[:-1])
+    for v in index:
+        params = msdki_params[v].copy()
+        params[1] = params[1] * params[0] ** 2


I think that at least this computation can be taken out of the loop:

msdki_params[:, 1] = msdki_params[:, 1] * msdki_params[: 0] ** 2 for ...

arokem · 2019-01-21T20:09:12Z

dipy/reconst/msdki.py

+            raise ValueError(e_s)
+
+        # Check if at least three b-values are given
+        enough_b = check_multi_b(self.gtab, 3, non_zero=False)


Shouldn't you pass bmag in here?

Yes! That is correct 👍

arokem · 2019-01-21T20:10:52Z

dipy/reconst/msdki.py

+               (Doctoral thesis). Downing College, University of Cambridge.
+               https://doi.org/10.17863/CAM.29356
+        """
+        return msdki_prediction(msdki_params, self.gtab, S0)


Things might go faster if msdki_prediction can use the mask.

arokem · 2019-01-21T20:14:24Z

doc/examples/reconst_msdki.py

+simulations (for more information on this type of simulations see
+:ref:`example_simulate_multi_tensor`). For this example, simulations are
+produced based on the sum of four diffusion tensors to represent the intra-
+and extra-cellular spaces of two fiber populations. The parameters of theses


"theses" => "these"

arokem · 2019-01-21T20:15:01Z

doc/examples/reconst_msdki.py

+
+"""
+For the acquisition parameters of the synthetic data, we use 60 gradient
+directions for two non-zero-b-values (1000 and 2000 $s/mm^{2}$) and two


"non-zero-b-values" => "non-zero b-values" (remove the dash between the two parts).

arokem · 2019-01-21T20:15:24Z

doc/examples/reconst_msdki.py

+"""
+For the acquisition parameters of the synthetic data, we use 60 gradient
+directions for two non-zero-b-values (1000 and 2000 $s/mm^{2}$) and two
+zero-bvalue (note that, such as the standard DKI, MSDKI requires at least


"zero-bvalue" => "zero b-values"

arokem · 2019-01-21T20:16:08Z

doc/examples/reconst_msdki.py

+gtab = gradient_table(bvals, bvecs)
+
+
+""" Simulations are lopped for different intra- and extra-cellular water


"lopped" => "looped"

arokem · 2019-01-21T20:16:51Z

doc/examples/reconst_msdki.py

+MD = dki_fit.md
+MK = dki_fit.mk(0, 3)
+
+""" Now we plot the results as a function of the ground truth interstection


"interstection" => "intersection"

ShreyasFadnavis · 2019-02-07T20:57:20Z

Hi @RafaelNH ! I am working on merging your PR! Apart from the minor changes that @arokem has suggested, LGTM! The code, tests and examples look super!

Can you please make the suggested changes by @arokem ?

arokem · 2019-02-15T20:47:37Z

Hey @RafaelNH : any chance you might be able to make these small fixes? This is almost ready to go!

…ude that values should be round function check_multi_b also uses this function to check the number of unique b-values added more tests to check both check_multi_b and round_bvals

…h b-values

…pherical DKI model File has a priliminary version of the names of the file's functions and object s

this file already have two simulations of spherical tensors for single and multi-voxels

function will has to be tested

… fit thanks Ariel Rokem

…re.gradients: function check_multi_b is now using this function

…t_tensor

in this way number of shells do not match number of parameters

…with DKI_micro)

…@skoudoro

…@skoudoro

RafaelNH · 2019-03-10T23:15:25Z

@arokem @ShreyasFadnavis ! Minor changes made! Sorry for the delay on this! I've improved the prediction function to avoid voxels looping, removed unnecessary changes in dti.py and added bmag optional input in maximum b-value requirement check!

skoudoro · 2019-03-11T02:41:29Z

Wow, Thanks for this compatibility update on commit ff68f5a!

I will wait until this Travis matrix succeed.

skoudoro · 2019-03-12T19:51:26Z

Thanks @RafaelNH for this nice work!

RafaelNH changed the title ~~(WIP) Mean Spherical DKI~~ (WIP) Mean Signal DKI Aug 31, 2017

nilgoyette reviewed May 1, 2018

View reviewed changes

arokem reviewed Jun 21, 2018

View reviewed changes

RafaelNH force-pushed the Mean_DKI branch from 9d942d6 to 5d1d3fb Compare January 9, 2019 18:15

RafaelNH changed the title ~~(WIP) Mean Signal DKI~~ Mean Signal DKI Jan 18, 2019

RafaelNH force-pushed the Mean_DKI branch from b93e42b to 32f8edd Compare January 18, 2019 16:57

skoudoro reviewed Jan 18, 2019

View reviewed changes

arokem mentioned this pull request Jan 21, 2019

Failure on Windows/Python 3.5 #1724

Closed

arokem reviewed Jan 21, 2019

View reviewed changes

skoudoro mentioned this pull request Mar 5, 2019

REL: 0.16.0 #1750

Merged

15 tasks

RafaelNH added 10 commits March 10, 2019 19:11

RF, TEST: add function that rounds b-values given the order of magnit…

498e50e

…ude that values should be round function check_multi_b also uses this function to check the number of unique b-values added more tests to check both check_multi_b and round_bvals

RF: free-water dti also uses check_multi_b to check if data was enoug…

711b40b

…h b-values

NF: Add new script that will contain the implementation of the mean s…

fd770c7

…pherical DKI model File has a priliminary version of the names of the file's functions and object s

TESTS: Add a file that will contain the tests for mdki

791f017

this file already have two simulations of spherical tensors for single and multi-voxels

TESTS, BF: Test MeanDiffusionKurtosisModel errors

4b0c36e

TESTS: Test if wrong mask is given to MDKI models fit

229379e

NF: Edit the helper function to fit the mean spherical dki

3d22660

function will has to be tested

RF: Edit dti's iter_fit_tensor in the way that it can be used by mdki…

9eb487b

… fit thanks Ariel Rokem

RF, NF, TESTS: Add function that gives the unique b-values in dipy.co…

3ad26c2

…re.gradients: function check_multi_b is now using this function

NF: Implement design_matrix for mdki

0ce7339

RafaelNH added 23 commits March 10, 2019 19:11

RF: ng as an optional parameter so that _wls_fit_mdki can use iter_fi…

30eadc3

…t_tensor

TESTS: Change mdki tests from 2 shell datasets to 3 shell datasets:

4c68c67

in this way number of shells do not match number of parameters

RF, BF: Working version of dki fit and start implementing predicition

2e04045

NF: Implementation of prediction functions

33373bc

DOC: correct typos in dipy.core.gradients.py

5cd8048

DOC: correct typos in gradient test

5d4aab5

DOC: Improve documentation on the core mdki files

40d6378

DOC: Minor corrections suggested by reviewers

6fa7e05

RF: Min signal set in constructor

8f0d3a4

TEST: Add tests to cover all code lines of mdki

d564c06

DOC: Start editing exemple for MSDKI

df7937c

DOC: Proofread msdki introduction'

ee4c2b8

DOC: Add sythetic signals in simulations

5f042aa

DOC: Finalize msdki example

552f223

STYLE: rename MDKI to MSDKI (to keep consistency and avoid confusion …

a375631

…with DKI_micro)

STYLE: rename inside files

013a62b

DOC: Add example in valid examples and examples_index.rst

151a9a6

DOC: Improve MSDKI example according to comments of @skoudoro

0dd4ff8

STYLE: Improvements on MSDKI's main code according to comments made by …

4be0628

…@skoudoro

RF: Vectorize prediction function

8cc1bc6

BW: Remove not required changes in dti

82751dd

DOC: more typos found

777b66b

BF: Add bmag in b-value maximum requirement check

792cad4

RafaelNH force-pushed the Mean_DKI branch from 1003bde to 792cad4 Compare March 10, 2019 23:11

Insure compatibility with python 2.7, numpy 1.7.1

ff68f5a

BF: Fixed bug on broadcasting S0 to 4D matrix

124f694

skoudoro merged commit ae74815 into dipy:master Mar 12, 2019

		gtab = gradient_table(bvals, bvecs)


		""" Simulations are lopped for different intra- and extra-cellular water

Mean Signal DKI #1230

Mean Signal DKI #1230

Conversation

RafaelNH commented Apr 22, 2017 • edited

skoudoro commented Apr 19, 2018

nilgoyette left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Jan 11, 2019 • edited

Codecov Report

RafaelNH commented Jan 18, 2019

arokem commented Jan 18, 2019

RafaelNH commented Jan 18, 2019

arokem commented Jan 18, 2019

RafaelNH commented Jan 18, 2019

skoudoro left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skoudoro Jan 18, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RafaelNH commented Jan 20, 2019

arokem left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShreyasFadnavis commented Feb 7, 2019 • edited

arokem commented Feb 15, 2019

RafaelNH commented Mar 10, 2019

skoudoro commented Mar 11, 2019

skoudoro commented Mar 12, 2019

RafaelNH commented Apr 22, 2017 •

edited

codecov-io commented Jan 11, 2019 •

edited

skoudoro Jan 18, 2019 •

edited

ShreyasFadnavis commented Feb 7, 2019 •

edited