WIP: Add mapmri flow #1350

aarya22 · 2017-10-07T18:53:19Z

Not sure how to create a workflow or if this is correct

@Garyfallidis could you check this out and see if this is what you meant?

Garyfallidis · 2017-10-08T01:55:01Z

Hi @Aarya, okay, you are moving towards the correct direction but not there yet. Will start commenting asap. Thanks!

aarya22 · 2017-10-11T21:47:59Z

@Garyfallidis anything I can change?

skoudoro

Hi @aarya22, it is a good start! I made some comment on your PR. Firstfall, you should do a git rebase on your code to update your code with the mainstream. Second, build test failed because of your matplotlib dependency (you can see it here. I suppose you do not really need this dependency. You can read the comment below to see why.
Let me know if you have any question or if you need help.

Thanks for this contribution!

skoudoro · 2017-10-12T02:33:22Z

dipy/workflows/reconst.py

+from dipy.reconst import mapmri
+import matplotlib
+matplotlib.use('agg')
+import matplotlib.pyplot as plt


I think, you do not need matplotlib here

skoudoro · 2017-10-12T02:35:17Z

dipy/workflows/reconst.py

+    def get_short_name(cls):
+        return 'mapmri'
+
+    def run(self, data_file, data_bvecs, data_bvals, small_delta, big_delta):


you should define output files init the run function. Look at the example on the following link : https://github.com/nipy/dipy/blob/master/dipy/workflows/reconst.py#L34

It will be good if you can add default value too. Indeed, it will be useful for beginners who do not know what to put.

skoudoro · 2017-10-12T02:37:16Z

dipy/workflows/reconst.py

+        return 'mapmri'
+
+    def run(self, data_file, data_bvecs, data_bvals, small_delta, big_delta):
+        img = nib.load(data_file)


try to use io_iteratorlike the example here. It permits you to read many input files.

skoudoro · 2017-10-12T02:37:34Z

dipy/workflows/reconst.py

+
+    def run(self, data_file, data_bvecs, data_bvals, small_delta, big_delta):
+        img = nib.load(data_file)
+        bvals,bvecs = read_bvals_bvecs(data_bval,data_bvec)


same as above

skoudoro · 2017-10-12T02:41:56Z

dipy/workflows/reconst.py

+        cax = divider.append_axes("right", size="5%", pad=0.05)
+        plt.colorbar(ind, cax=cax)
+
+        plt.savefig('MAPMRI_maps_regularization.png')


I suppose we do not want to save this data in this way but we prefer to return many output files that can be input files for another process (or workflow). Currently, we can not do that with your code. So you should remove the matplolib part and replace it. you can see an example in the link above

skoudoro · 2017-10-12T02:45:34Z

dipy/workflows/reconst.py

+        return 'mapmri'
+
+    def run(self, data_file, data_bvecs, data_bvals, small_delta, big_delta):
+        img = nib.load(data_file)


Documentation is really important here. You can look at the example here how to make a good documentation

codecov-io · 2017-10-31T06:19:19Z

Codecov Report

Merging #1350 into master will decrease coverage by <.01%.
The diff coverage is 83.78%.

@@            Coverage Diff             @@
##           master    #1350      +/-   ##
==========================================
- Coverage   87.07%   87.06%   -0.01%     
==========================================
  Files         227      228       +1     
  Lines       29060    29170     +110     
  Branches     3123     3136      +13     
==========================================
+ Hits        25303    25398      +95     
- Misses       3047     3051       +4     
- Partials      710      721      +11

Impacted Files	Coverage Δ
dipy/workflows/tests/test_reconst_dti.py	`95.08% <ø> (ø)`	⬆️
dipy/workflows/reconst.py	`79.38% <79.41%> (-0.12%)`	⬇️
dipy/workflows/tests/test_reconst_mapmri.py	`90.69% <90.69%> (ø)`
dipy/reconst/mapmri.py	`89.78% <0%> (+0.38%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e34445c...af4cd7d. Read the comment docs.

aarya22 · 2017-10-31T06:24:22Z

Sorry for the long delay in replying to your comments @skoudoro I have been busy with school and midterms. Take a look at the updated workflow. Unfortunately, I cannot change the output of the workflow. Brain-Life is planning on using this workflow to power one of its applications (app-dipy-mapmri) and needs the png file as its output in order to work. I am also not sure what else to change it to so it can be compatible with other dipy workflows. Let me know if there are any problems here and hopefully I can change it promptly.

skoudoro

no problem and thanks for that @aarya22! I made some few comments below.

Can you add a test for your workflow? you will find an example in dipy/workflows/tests/test_reconst_dti

I am surprised that this workflow output is a plot screenshot and I wonder How you can use it as an input of other brain-life apps? Can you explain me more?

skoudoro · 2017-10-31T17:35:35Z

dipy/workflows/reconst.py

+            Path to the input volume.
+        data_bvecs : string
+            Path to the bvec files.
+        data_bvals : 


add the datatype -> data_bvals : string

skoudoro · 2017-10-31T17:36:12Z

dipy/workflows/reconst.py

+            Path to the bvec files.
+        data_bvals : 
+            Path to the bval files.
+        small_delta :


add the datatype -> small_delta : float

skoudoro · 2017-10-31T17:36:40Z

dipy/workflows/reconst.py

+        small_delta :
+            Small delta value used in generation of gradient table of provided
+            bval and bvec. (default: 0.0129)
+        big_delta :


add the datatype -> big_delta : float

skoudoro · 2017-10-31T17:38:31Z

dipy/workflows/reconst.py

+        big_delta :
+            Big delta value used in generation of gradient table of provided 
+            bval and bvec. (default: 0.0218)
+        save_metrics :


add the datatype -> save_metric : string

skoudoro · 2017-10-31T17:42:19Z

dipy/workflows/reconst.py

+                                big_delta=big_delta, b0_threshold=50)
+
+
+            data_small = data[60:85, 80:81, 60:85]


Why do you crop the data? if it is really necessary, you have to warn the user that only a small part of its data will be processed. Otherwise, use the whole data.

arokem · 2017-10-31T20:06:54Z

I would not make the output of this a png. If brain-life requires a png, I would do this in the definition of the app there.

…

On Tue, Oct 31, 2017 at 11:00 AM, Serge Koudoro ***@***.***> wrote: ***@***.**** requested changes on this pull request. no problem and thanks for that @aarya22 <https://github.com/aarya22>! I made some few comments below. Can you add a test for your workflow? you will find an example in dipy/workflows/tests/test_reconst_dti <https://github.com/nipy/dipy/blob/master/dipy/workflows/tests/test_reconst_dti.py> I am surprised that this workflow output is a plot screenshot and I wonder How you can use it as an input of other brain-life apps? Can you explain me more? ------------------------------ In dipy/workflows/reconst.py <#1350 (comment)>: > + return 'mapmri' + + def run(self, data_file, data_bvecs, data_bvals, small_delta=0.0129, big_delta=0.0218 + save_metrics = [], out_dir='', out_mapmri='MAPMRI_maps_regularization.png'): + """ Workflow for the app-dipy-mapmri on Brain-Life (www.brain-life.org). + Generates RTOP graphs saved in a .png format in input files provided by + `data_file` and saves the png file to an output directory specified by + `out_dir`. + + Parameters + ---------- + data_file : string + Path to the input volume. + data_bvecs : string + Path to the bvec files. + data_bvals : add the datatype -> data_bvals : string ------------------------------ In dipy/workflows/reconst.py <#1350 (comment)>: > + def run(self, data_file, data_bvecs, data_bvals, small_delta=0.0129, big_delta=0.0218 + save_metrics = [], out_dir='', out_mapmri='MAPMRI_maps_regularization.png'): + """ Workflow for the app-dipy-mapmri on Brain-Life (www.brain-life.org). + Generates RTOP graphs saved in a .png format in input files provided by + `data_file` and saves the png file to an output directory specified by + `out_dir`. + + Parameters + ---------- + data_file : string + Path to the input volume. + data_bvecs : string + Path to the bvec files. + data_bvals : + Path to the bval files. + small_delta : add the datatype -> small_delta : float ------------------------------ In dipy/workflows/reconst.py <#1350 (comment)>: > + Generates RTOP graphs saved in a .png format in input files provided by + `data_file` and saves the png file to an output directory specified by + `out_dir`. + + Parameters + ---------- + data_file : string + Path to the input volume. + data_bvecs : string + Path to the bvec files. + data_bvals : + Path to the bval files. + small_delta : + Small delta value used in generation of gradient table of provided + bval and bvec. (default: 0.0129) + big_delta : add the datatype -> big_delta : float ------------------------------ In dipy/workflows/reconst.py <#1350 (comment)>: > + + Parameters + ---------- + data_file : string + Path to the input volume. + data_bvecs : string + Path to the bvec files. + data_bvals : + Path to the bval files. + small_delta : + Small delta value used in generation of gradient table of provided + bval and bvec. (default: 0.0129) + big_delta : + Big delta value used in generation of gradient table of provided + bval and bvec. (default: 0.0218) + save_metrics : add the datatype -> save_metric : string ------------------------------ In dipy/workflows/reconst.py <#1350 (comment)>: > + """ + io_it = self.get_io_iterator() + for dwi, bval, bvec, mapname in io_it: + + logging.info('Computing DTI metrics for {0}'.format(dwi)) + img = nib.load(dwi) + data = img.get_data() + bvals,bvecs = read_bvals_bvecs(bval, bvec) + + + gtab = gradient_table(bvals=bvals, bvecs=bvecs, + small_delta=small_delta, + big_delta=big_delta, b0_threshold=50) + + + data_small = data[60:85, 80:81, 60:85] Why do you crop the data? if it is really necessary, you have to warn the user that only a small part of its data will be processed. Otherwise, use the whole data. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1350 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHPNj667zJduuOuFuCWklQV1klVj5o6ks5sx2BFgaJpZM4PxdHu> .

aarya22 · 2017-11-02T17:09:48Z

The reason I cropped the data and had the output be a png file is because I wanted this to be a simple preliminary test for making a brain-life application from a dipy workflow. I was thinking that eventually brain-life applications can simply become wrappers for the dipy workflow.

I agree that png files should not be an output of the workflow, but I am not sure what else I should return from Mapmir. I do not know enough about the model, if you have any suggestions I will gladly take them.

arokem · 2017-11-02T17:22:31Z

We can talk about that when we meet tomorrow. Ideally, this would produce something similar to what the tensor workflow produces ( https://github.com/nipy/dipy/blob/master/dipy/workflows/reconst.py#L25) -- nifti files that store model parameters and derived maps.

…

On Thu, Nov 2, 2017 at 10:09 AM, Aman Arya ***@***.***> wrote: The reason I cropped the data and had the output be a png file is because I wanted this to be a simple preliminary test for making a brain-life application from a dipy workflow. I was thinking that eventually brain-life applications can simply become wrappers for the dipy workflow. I agree that png files should not be an output of the workflow, but I am not sure what else I should return from Mapmir. I do not know enough about the model, if you have any suggestions I will gladly take them. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1350 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHPNoQvuYpdr8Ol_rEwgJnZ0v53v35nks5syfddgaJpZM4PxdHu> .

Mmriflow

Fix unit tests

…kflow .

Mapmri workflow

aarya22 · 2017-12-08T02:11:01Z

@skoudoro @arokem I've managed to get the Travis build test work for most of the environments, but there are a few environments that do not provide the CVXPY package which is necessary for the workflow to finish. I'm not sure how to access the config so I can include the CVXPY package in all the environments.

arokem · 2017-12-08T02:39:16Z

Instead of installing cvxpy on all the machines, skip the tests that require cvxpy, if that's not installed: Like here: https://github.com/nipy/dipy/blob/master/dipy/reconst/tests/test_mapmri.py#L624

…

On Thu, Dec 7, 2017 at 6:11 PM, Aman Arya ***@***.***> wrote: @skoudoro <https://github.com/skoudoro> @arokem <https://github.com/arokem> I've managed to get the Travis build test work for most of the environments, but there are a few environments that do not provide the CVXPY package which is necessary for the workflow to finish. I'm not sure how to access the config so I can include the CVXPY package in all the environments. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1350 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHPNpS-vdvWaSfnkB6-cr4rm1_lBWoGks5s-Jq1gaJpZM4PxdHu> .

Add decorators to remove cvxpy from testing

aarya22 · 2017-12-10T04:02:32Z

@skoudoro @arokem Travis has passed, but I'm not sure what the other two errors are about or what I am supposed to be changing there.

arokem

A few smaller comments, and questions about the overall design. @Garyfallidis : could you please take a look at this. Are we really supposed to have a separate object for every option of the model? That is a separate object for Laplacian regularization, for the positivity constraint, and for both?

arokem · 2018-01-01T16:57:24Z

dipy/workflows/reconst.py

+        return 'mapmri'
+
+    def run(self, data_file, data_bvecs, data_bvals, model_type='', out_rtop='rtop.nii.gz',
+            out_lapnorm='lapnorm.nii.gz',out_msd='msd.nii.gz', out_qiv='qiv.nii.gz', out_rtap='rtap.nii.gz',


PEP8: this line is >80 characters

arokem · 2018-01-01T16:57:39Z

dipy/workflows/reconst.py

+            out_lapnorm='lapnorm.nii.gz',out_msd='msd.nii.gz', out_qiv='qiv.nii.gz', out_rtap='rtap.nii.gz',
+            out_rtpp='rtpp.nii.gz', small_delta=0.0129, big_delta=0.0218, save_metrics=[], out_dir=''):
+        """ Workflow for the app-dipy-mapmri on Brain-Life (www.brain-life.org). 
+        Generates rtop, lapnorm, msd, qiv, rtap, rtpp saved in a nifti format in input files provided by


PEP8: line is >80 characters

arokem · 2018-01-01T16:57:56Z

dipy/workflows/reconst.py

+        io_it = self.get_io_iterator()
+        for dwi, bval, bvec, out_rtop, out_lapnorm, out_msd, out_qiv, out_rtap, out_rtpp in io_it:
+
+            logging.info('Computing DTI metrics for {0}'.format(dwi))


This should be "Computing MAPMRI metrics..."

arokem · 2018-01-01T16:59:11Z

dipy/workflows/reconst.py

+    def get_short_name(cls):
+        return 'mapmri'
+
+    def run(self, data_file, data_bvecs, data_bvals, model_type='', out_rtop='rtop.nii.gz',


Instead of model_type, I would suggest having a keyword argument for "laplacian" and for "positivity" (both set to True per default)

And a keyword argument for the laplacian weighting as well.

arokem · 2018-01-01T17:01:33Z

dipy/workflows/reconst.py

+                                                     positivity_constraint=True)
+                mapfit_aniso = map_model_aniso.fit(data)
+
+            # Not sure where to get affine or metadata?


Is this still an issue? Getting the affine? You can get it from the data_file.

As an aside: if you have questions, it's better to ask them in the PR interface than put them as comments in your code!

Ah thanks for pointing that out. I added those notes in the code to ask you at our weekly sessions and forgot to delete them.

arokem · 2018-01-01T17:02:59Z

dipy/workflows/reconst.py

+                         format(os.path.dirname(out_rtpp)))
+
+
+class ReconstMAPMRILaplacian(ReconstMAPMRIFlow):


@Garyfallidis : is this the right way to do things? You create an interface for every setting of options to the model? I thought this was supposed to create a CLI that allows you to input key-word arguments?

arokem · 2018-01-03T23:44:59Z

Closed in favor of #1395

Add mapmri flow

b680d37

aarya22 mentioned this pull request Oct 7, 2017

"CVXOPT package needed to enforce constraints" brainlife/app-dipy-MAPMRI#1

Open

skoudoro reviewed Oct 12, 2017

View reviewed changes

Edit mapmri workflows

6784b17

skoudoro requested changes Oct 31, 2017

View reviewed changes

aarya22 added 2 commits November 21, 2017 18:13

Add mapmri flow

7324b36

Edit mapmri workflows

61f7729

aarya22 force-pushed the master branch from 6784b17 to 61f7729 Compare November 22, 2017 02:15

aarya22 added 4 commits November 22, 2017 00:55

Add to ReconstMAPMRI and skeleton of test

7fdaca8

Add mapmri flow

ad71f3c

Edit mapmri workflows

f1e2fac

Add to ReconstMAPMRI and skeleton of test

1ac2efb

aarya22 changed the title ~~Add mapmri flow~~ WIP: Add mapmri flow Nov 26, 2017

aarya22 added 10 commits November 26, 2017 23:51

Wrote test workflows and reconst workflows

41f6fa5

Wrote test workflows and reconst workflows

7b681ad

Merge branch 'master' into mmriflow

e112c7d

Merge pull request #1 from aarya22/mmriflow

def4a5e

Mmriflow

Add to main mapmri iterator and fix args issue

cf1486d

Add nifti file saves and create nifti img

63237b2

Change to np.float32 type

28eca10

Fix last gen outputs

ba27cd6

Remove debugging

b29a713

Merge pull request #2 from aarya22/MAPMRI_Workflow

dd02b92

Fix unit tests

aarya22 added 3 commits December 7, 2017 10:53

Merge branch 'master' of https://github.com/nipy/dipy into MAPMRI_Wor…

a9f3865

…kflow .

Fix unit tests

7bdff17

Merge pull request #3 from aarya22/MAPMRI_Workflow

47dc497

Mapmri workflow

arokem mentioned this pull request Dec 7, 2017

Mapmri workflow #1388

Closed

aarya22 added 2 commits December 9, 2017 18:30

Add ignore to tests that do not have cxvpy

b0c204a

Merge pull request #4 from aarya22/MAPMRI_Workflow

af4cd7d

Add decorators to remove cvxpy from testing

arokem mentioned this pull request Jan 1, 2018

DataType and input files brainlife/app-dipy-MAPMRI#3

Open

arokem requested changes Jan 1, 2018

View reviewed changes

aarya22 mentioned this pull request Jan 3, 2018

Mapmri workflow #1395

Closed

arokem closed this Jan 3, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Add mapmri flow #1350

WIP: Add mapmri flow #1350

aarya22 commented Oct 7, 2017

Garyfallidis commented Oct 8, 2017

aarya22 commented Oct 11, 2017

skoudoro left a comment

skoudoro Oct 12, 2017

skoudoro Oct 12, 2017

skoudoro Oct 12, 2017

skoudoro Oct 12, 2017

skoudoro Oct 12, 2017

skoudoro Oct 12, 2017

codecov-io commented Oct 31, 2017 •

edited

Loading

aarya22 commented Oct 31, 2017

skoudoro left a comment

skoudoro Oct 31, 2017

skoudoro Oct 31, 2017

skoudoro Oct 31, 2017

skoudoro Oct 31, 2017

skoudoro Oct 31, 2017

arokem commented Oct 31, 2017 via email

aarya22 commented Nov 2, 2017

arokem commented Nov 2, 2017 via email

aarya22 commented Dec 8, 2017

arokem commented Dec 8, 2017 via email

aarya22 commented Dec 10, 2017

arokem left a comment

arokem Jan 1, 2018

arokem Jan 1, 2018

arokem Jan 1, 2018

arokem Jan 1, 2018

arokem Jan 1, 2018

arokem Jan 1, 2018

aarya22 Jan 2, 2018

arokem Jan 1, 2018

arokem commented Jan 3, 2018

		big_delta=big_delta, b0_threshold=50)


		data_small = data[60:85, 80:81, 60:85]

		format(os.path.dirname(out_rtpp)))


		class ReconstMAPMRILaplacian(ReconstMAPMRIFlow):

WIP: Add mapmri flow #1350

WIP: Add mapmri flow #1350

Conversation

aarya22 commented Oct 7, 2017

Garyfallidis commented Oct 8, 2017

aarya22 commented Oct 11, 2017

skoudoro left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Oct 31, 2017 • edited Loading

Codecov Report

aarya22 commented Oct 31, 2017

skoudoro left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arokem commented Oct 31, 2017 via email

aarya22 commented Nov 2, 2017

arokem commented Nov 2, 2017 via email

aarya22 commented Dec 8, 2017

arokem commented Dec 8, 2017 via email

aarya22 commented Dec 10, 2017

arokem left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arokem commented Jan 3, 2018

codecov-io commented Oct 31, 2017 •

edited

Loading