Updated docs for using P2S optimally #2382

ShreyasFadnavis · 2021-05-01T11:45:37Z

Added information about parameters of Patch2Self
Also added information on applying Patch2Self on batches

ShreyasFadnavis · 2021-05-01T11:54:52Z

Hi! Sorry for the slowness in this small PR. I have added information about how to use Patch2Self optimally. It is ready to be reviewed 👍🏽

arokem · 2021-05-01T13:26:29Z

doc/examples/denoise_patch2self.py

+ridge regression (model='ridge').
+
+*Please do note that sometimes using ridge regression can hamper the
+performance of Patch2Self. If so, please use model='ols'.*


Suggested change

performance of Patch2Self. If so, please use model='ols'.*

performance of Patch2Self. If so, please use model='ols'.

arokem · 2021-05-01T13:29:16Z

doc/examples/denoise_patch2self.py

+After doing this, the 2 denoised batches can be merged as follows:
+`denoised_p2s = np.concatenate((denoised_batch1, denoised_batch2), axis=3)`
+
+If using Patch2Self for research purposes, acquisition design tests, etc., one


Suggested change

If using Patch2Self for research purposes, acquisition design tests, etc., one

If using Patch2Self for research purposes, acquisition design tests, etc., one

I am not sure what the "research purposes" means here. The following "acquisition design, tests, etc." suggests that this is not something that users should do if they want to analyze the data? But I am not sure.

Yes! Agreed... Taking that part of the line off. I just wanted to encourage people to experiment with the algorithm.

arokem · 2021-05-01T13:31:08Z

doc/examples/denoise_patch2self.py

-denoised_arr = patch2self(data, bvals, shift_intensity=True,
-                          clip_negative_vals=False)
+denoised_arr = patch2self(data, bvals, model='ols', shift_intensity=True,
+                          clip_negative_vals=False, b0_threshold=100)


Sorry -- one more: why is b0_threshold set to a higher value than the gradient_table default?

Okay... Set it back to 50, which is the default (even in the codebase). I added a note that people should change it if they are working with data such as HCP 👍🏽

arokem · 2021-05-03T02:38:03Z

doc/examples/denoise_patch2self.py

+
+The default value of `b0_threshold` in DIPY is set to 50. If using data
+such as HCP, the b0 volume tends to have a higher value, please set it to 100
+or more accordingly.


Hmmm. The b-values that were used in the HCP 3T dataset are:

array([ 5., 990., 995., 1000., 1005., 1985., 1990., 1995., 2000., 2005., 2010., 2980., 2985., 2990., 2995., 3000., 3005., 3010.])

So, setting b0_threshold=50 and setting b0_threshold=100 would presumably give you exactly the same thing. I am also confused about the phrase the b0 volume tends to have a higher value -- it doesn't seem like this parameter would interact with the values of the image, only with the selection of the image volumes. Or am I missing something?

Hi @arokem -- That is correct. If the bvals are less than 50 the b0_threshold at 50 and 100 would behave exactly the same way. Here are the bvals for HCP 7T for instance:

As you can see they are >=50 somewhere between 55 and 65. Although not common, I do want to warn users in case they have such datasets. Let me try rephrasing the sentence to make it clearer.

arokem · 2021-05-03T12:23:23Z

doc/examples/denoise_patch2self.py

-denoised_arr = patch2self(data, bvals, shift_intensity=True,
-                          clip_negative_vals=False)
+denoised_arr = patch2self(data, bvals, model='ols', shift_intensity=True,
+                          clip_negative_vals=False, b0_denoising=50)


Suggested change

clip_negative_vals=False, b0_denoising=50)

clip_negative_vals=False, b0_threshold=50)

More here: I believe that b0_denoising is a boolean.

Or possibly

Suggested change

clip_negative_vals=False, b0_denoising=50)

clip_negative_vals=False, b0_denoising=True)

But both are the default values, so it's not clear why you are setting either of them.

My bad! Good catch -- fixing this.

arokem · 2021-05-03T12:26:08Z

doc/examples/denoise_patch2self.py


 """
+The above parameters should give optimal denoising performance for Patch2Self.


Optimal in this dataset, or generally? If generally, why are these not the default values? For example, the warning below about ridge hampering performance is rather ominous. Should we change the default to 'ols'?

Optimal in this dataset, or generally?

'ols' is optimal generally. Same goes for shift_intensity=True and clip_negative_vals=False 👍🏽 Ridge does seem to be useful to speedup the regression. I was also looking into using the regression module of sklearn and found this comment from the discussion very useful: scikit-learn/scikit-learn#13923 (comment).

Should I use this? I was thinking of adding an option called ols_fast instead of just replacing ols with ridge and a very small alpha. What do you think?

I think that it's very odd that ridge is faster than OLS, given that they essentially are the same algorithm with a small tweak that shouldn't really affect performance at all. But 🤷‍♂️

Yep... me too!😂

arokem · 2021-05-03T12:26:32Z

doc/examples/denoise_patch2self.py

@@ -82,6 +90,15 @@
 Please set `shift_intensity=True` and `clip_negative_vals=False` by default to
 avoid negative values in the denoised output.

+The `b0_threshold` is used to separate the b0 volumes from the DWI volumes.
+Changing the value of the b0 threshold is needed is the b0 volumes in the


Suggested change

Changing the value of the b0 threshold is needed is the b0 volumes in the

Changing the value of the b0 threshold is needed if the b0 volumes in the

arokem · 2021-05-03T12:26:54Z

doc/examples/denoise_patch2self.py

+
+The default value of `b0_threshold` in DIPY is set to 50. If using data
+such as HCP 7T, the b0 volumes tend to have a higher b-value (>=50)
+assoiciated with them in the `bval` file. Please check the b-values for b0s and


Suggested change

assoiciated with them in the `bval` file. Please check the b-values for b0s and

associated with them in the `bval` file. Please check the b-values for b0s and

ShreyasFadnavis · 2021-05-03T14:26:35Z

While we are at it: should I also add Patch2Self in every tutorial as a primary step? I remember @arokem suggesting this in a previous PR 👍🏽 P2S is fast... adds ~3-5 mins for Stanford HARDI data per example (I think used most pervasively). Is this okay? Or will it drive up the compilation time of all tutorials drastically?

skoudoro · 2021-05-03T14:32:05Z

70 tutorials x 5min = 350min =~ 6h more.... so I recommend to pick up not more than 10 tutorials.

ShreyasFadnavis · 2021-05-03T14:47:41Z

70 tutorials x 5min = 350min =~ 6h more.... so I recommend to pick up not more than 10 tutorials.

Yes! I don't think it's relevant for all tutorials either :)

Here are the ones where I would add it:

CSD and MCSD
3D SHORE
MAP-MRI
DTI
DKI
FW-DTI
WMTI
Basic Tracking EuDX
Deterministic Tracking
Probabilistic Tracking
Particle Filter Tracking

Does this sound okay?

skoudoro · 2021-05-03T14:53:38Z

Does this sound okay?

+1 👍🏾

codecov · 2021-05-03T16:34:40Z

Codecov Report

Merging #2382 (f2d028a) into master (4148983) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master    #2382   +/-   ##
=======================================
  Coverage   85.23%   85.24%           
=======================================
  Files         126      126           
  Lines       16575    16575           
  Branches     2686     2686           
=======================================
+ Hits        14128    14129    +1     
+ Misses       1762     1759    -3     
- Partials      685      687    +2

Impacted Files	Coverage Δ
dipy/denoise/patch2self.py	`73.63% <100.00%> (ø)`
dipy/viz/app.py	`77.01% <0.00%> (-0.49%)`	⬇️
dipy/reconst/forecast.py	`89.84% <0.00%> (+1.52%)`	⬆️

skoudoro · 2021-05-03T23:11:40Z

@arokem, Do you want to have a last look? What is the final decision concerning this PR?

arokem · 2021-05-04T02:46:52Z

I would only change those tutorials that already have denoising in them, but I think that we can take that on another PR (or even better, series of PRs). I think that this PR is ready to go.

skoudoro · 2021-05-04T20:53:55Z

Thank you @ShreyasFadnavis, thank you for the review @arokem

ShreyasFadnavis added 2 commits May 1, 2021 07:42

updated docs for using P2S optimally

47ec57c

updated b0_threshold info

4034b44

arokem reviewed May 1, 2021

View reviewed changes

ShreyasFadnavis added 3 commits May 2, 2021 13:12

addressed Ariel's comments

5905128

minor fix

8eda30e

addressed 1 missing comment

15b6e2b

arokem reviewed May 3, 2021

View reviewed changes

address Ariel's comment

7982997

arokem reviewed May 3, 2021

View reviewed changes

ShreyasFadnavis added 3 commits May 3, 2021 08:40

fixing as per Ariel's suggestion

def815a

spellcheck

50cf7ab

spellcheck

f2d028a

skoudoro added this to the 1.4.1 milestone May 3, 2021

skoudoro added the 📓type:Documentation 📓 label May 3, 2021

skoudoro merged commit 00af530 into dipy:master May 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updated docs for using P2S optimally #2382

Updated docs for using P2S optimally #2382

ShreyasFadnavis commented May 1, 2021 •

edited

Loading

ShreyasFadnavis commented May 1, 2021

arokem May 1, 2021

arokem May 1, 2021

ShreyasFadnavis May 2, 2021

arokem May 1, 2021

ShreyasFadnavis May 2, 2021

arokem May 3, 2021

ShreyasFadnavis May 3, 2021 •

edited

Loading

arokem May 3, 2021

arokem May 3, 2021

ShreyasFadnavis May 3, 2021

arokem May 3, 2021

ShreyasFadnavis May 3, 2021

arokem May 3, 2021

ShreyasFadnavis May 3, 2021 •

edited

Loading

arokem May 4, 2021

ShreyasFadnavis May 4, 2021

arokem May 3, 2021

arokem May 3, 2021

ShreyasFadnavis commented May 3, 2021 •

edited

Loading

skoudoro commented May 3, 2021

ShreyasFadnavis commented May 3, 2021

skoudoro commented May 3, 2021

codecov bot commented May 3, 2021

skoudoro commented May 3, 2021

arokem commented May 4, 2021

skoudoro commented May 4, 2021

	performance of Patch2Self. If so, please use model='ols'.*
	performance of Patch2Self. If so, please use model='ols'.

	If using Patch2Self for research purposes, acquisition design tests, etc., one
	If using Patch2Self for research purposes, acquisition design tests, etc., one

	clip_negative_vals=False, b0_denoising=50)
	clip_negative_vals=False, b0_threshold=50)

	clip_negative_vals=False, b0_denoising=50)
	clip_negative_vals=False, b0_denoising=True)


		"""
		The above parameters should give optimal denoising performance for Patch2Self.

	Changing the value of the b0 threshold is needed is the b0 volumes in the
	Changing the value of the b0 threshold is needed if the b0 volumes in the

	assoiciated with them in the `bval` file. Please check the b-values for b0s and
	associated with them in the `bval` file. Please check the b-values for b0s and

Updated docs for using P2S optimally #2382

Updated docs for using P2S optimally #2382

Conversation

ShreyasFadnavis commented May 1, 2021 • edited Loading

ShreyasFadnavis commented May 1, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShreyasFadnavis May 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShreyasFadnavis May 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShreyasFadnavis commented May 3, 2021 • edited Loading

skoudoro commented May 3, 2021

ShreyasFadnavis commented May 3, 2021

skoudoro commented May 3, 2021

codecov bot commented May 3, 2021

Codecov Report

skoudoro commented May 3, 2021

arokem commented May 4, 2021

skoudoro commented May 4, 2021

ShreyasFadnavis commented May 1, 2021 •

edited

Loading

ShreyasFadnavis May 3, 2021 •

edited

Loading

ShreyasFadnavis May 3, 2021 •

edited

Loading

ShreyasFadnavis commented May 3, 2021 •

edited

Loading