Add FAQ, Tips, Tricks section to RTD #1601

franklin-feingold · 2019-05-01T23:38:27Z

Changes proposed in this pull request

Added a FAQ, Tips, Tricks section to RTD. This initial PR is to initalize this section and transfer the Wiki over to the RTD. Future PRs will address specific documentation issues to be added to this section.

Documentation that should be reviewed

Added FAQ, Tips, Tricks section to RTD

update 5/1

effigies · 2019-05-02T17:13:06Z

@franklin-feingold I cancelled the get*_data jobs. This only needs to pass the documentation build. In the future, if you name your branches with the prefix doc/, only the jobs needed to run the doc build will be run on Circle, saving a lot of time. (No need to change it now.)

emdupre · 2019-05-02T19:03:14Z

docs/faq_tips_tricks.rst

+
+**My ``FMRIPREP`` run is hanging...**
+
+There is a Python bug that affects ``FMRIPREP`` when processes are killed for running out of memory. While we are working on finding a solution that does not run up against this bug, this may take some time. This can be most easily resolved by allocating more memory to the process, if possible. Additionally, consider using the ``--low-mem`` flag, which will make some memory optimizations at the cost of disk space in the working directory.


Should we include information here on how to increase the memory allocation to a Docker container ?

Also, very nit-picky request -- would it be possible to put each sentence on a newline ?

This PR was to more so initialize this page and move over the wiki. To breakdown the PRs I will put that into another (in addition to #1577, #1573, #1570)

Also committed newlining each new sentence

effigies · 2019-05-02T23:46:46Z

docs/index.rst

@@ -22,3 +22,4 @@ Contents
   contributors
   citing
   api/index.rst
+   faq_tips_tricks


Can you move this between outputs and contributors?

effigies · 2019-05-02T23:50:25Z

docs/faq_tips_tricks.rst

@@ -0,0 +1,31 @@
+.. include:: links.rst
+
+FAQ, Tips, and Tricks


To be consistent with the other pages.

Suggested change

FAQ, Tips, and Tricks

=====================

FAQ, Tips, and Tricks

effigies · 2019-05-02T23:50:35Z

docs/faq_tips_tricks.rst

+.. include:: links.rst
+
+FAQ, Tips, and Tricks
+------------------------------


Suggested change

------------------------------

=====================

effigies · 2019-05-02T23:52:01Z

docs/faq_tips_tricks.rst

+FAQ, Tips, and Tricks
+------------------------------
+
+**Should I run quality control of my images before running ``FMRIPREP``?**


Here's the rendered result: https://8229-53175327-gh.circle-artifacts.com/0/home/circleci/out/docs/faq_tips_tricks.html

The back-ticks don't seem to work in bold. Also, if we make this a sub-heading, then it will be linkable.

Suggested change

**Should I run quality control of my images before running ``FMRIPREP``?**

Should I run quality control of my images before running fMRIPrep?

------------------------------------------------------------------

effigies · 2019-05-02T23:53:21Z

docs/faq_tips_tricks.rst

+When using publicly available datasets, an additional concern is that images may have gone through some kind of preprocessing (see next question).
+
+
+**What if I find some images have undergone some pre-processing already (e.g. my T1w image is already skull-stripped)?**


I'm not going to try to match the number of characters on this one.

Suggested change

**What if I find some images have undergone some pre-processing already (e.g. my T1w image is already skull-stripped)?**

What if I find some images have undergone some pre-processing already (e.g., my T1w image is already skull-stripped)?

----------------------------------------------------

effigies · 2019-05-02T23:54:12Z

docs/faq_tips_tricks.rst

+**What if I find some images have undergone some pre-processing already (e.g. my T1w image is already skull-stripped)?**
+
+These images imply an unknown level of preprocessing (e.g. was it already bias-field corrected?), which makes it difficult to decide on best-practices for further processing.
+Hence, supporting such images was considered very low priority for ``FMRIPREP``.


I think we've generally moved to this format. We should try to be consistent across pages, so maybe have a look before accepting this one.

Suggested change

Hence, supporting such images was considered very low priority for ``FMRIPREP``.

Hence, supporting such images was considered very low priority for fMRIPrep.

effigies · 2019-05-02T23:54:39Z

docs/faq_tips_tricks.rst

+So for OpenFMRI, we've been excluding these subjects, and for user-supplied data, we would recommend reverting to the original, defaced, T1w images to ensure more uniform preprocessing.
+
+
+**My ``FMRIPREP`` run is hanging...**


Suggested change

**My ``FMRIPREP`` run is hanging...**

My fMRIPrep run is hanging...

-----------------------------

effigies · 2019-05-02T23:54:56Z

docs/faq_tips_tricks.rst

+
+**My ``FMRIPREP`` run is hanging...**
+
+There is a Python bug that affects ``FMRIPREP`` when processes are killed for running out of memory. 


Suggested change

There is a Python bug that affects ``FMRIPREP`` when processes are killed for running out of memory.

There is a Python bug that affects fMRIPrep when processes are killed for running out of memory.

moved faq_tips_tricks to between outputs and contributors

franklin-feingold · 2019-05-03T00:16:42Z

@effigies thank you for your review! addressed your comments in recent commits

oesteban

Left some suggestions...

oesteban · 2019-05-02T02:37:28Z

docs/faq_tips_tricks.rst

+FAQ, Tips, and Tricks
+-----
+
+**Should I run quality control of my images before running ``FMRIPREP``?**


Suggested change

**Should I run quality control of my images before running ``FMRIPREP``?**

**Should I run quality control of my images before running *fMRIPrep*?**

oesteban · 2019-05-03T18:39:13Z

docs/faq_tips_tricks.rst

+Should I run quality control of my images before running fMRIPrep?
+--------------------------------------------------------------------
+
+Yes. You should as well before any other processing/analysis.


Suggested change

Yes. You should as well before any other processing/analysis.

Yes. You should do so before any processing/analysis takes place.

oesteban · 2019-05-03T18:39:44Z

docs/faq_tips_tricks.rst

+Yes. You should as well before any other processing/analysis.
+
+Oftentimes (more often than we would like), images have fatal artifacts and problems. 
+Those would not pass any QC check.


Suggested change

Those would not pass any QC check.

oesteban · 2019-05-03T18:46:21Z

docs/faq_tips_tricks.rst

+
+Oftentimes (more often than we would like), images have fatal artifacts and problems. 
+Those would not pass any QC check.
+Some other times, QC checks will flag some images that we should carefully track throughout processing.


Suggested change

Some other times, QC checks will flag some images that we should carefully track throughout processing.

Some exclusion criteria for data quality should be pre-specified before QC and any screening

of the original data.

Those exclusion criteria must be designed in agreement with the goals and challenges of the

experimental design.

For instance, when it is planned to run some cortical thickness analysis, images should be excluded

even when they present the most subtle ghosts or other artifacts that may introduce biases in surface

reconstruction.

However, if the same artifactual data was planned to be used just as a reference for spatial

normalization, some of those artifacts should be noted, but may not grant exclusion of the data.

oesteban · 2019-05-03T18:49:45Z

docs/faq_tips_tricks.rst

+
+
+My fMRIPrep run is hanging...
+---------------------------------


Suggested change

---------------------------------

-----------------------------

oesteban · 2019-05-03T18:53:45Z

docs/faq_tips_tricks.rst

+My fMRIPrep run is hanging...
+---------------------------------
+
+There is a Python bug that affects fMRIPrep when processes are killed for running out of memory. 


Suggested change

There is a Python bug that affects fMRIPrep when processes are killed for running out of memory.

When running on Linux platforms (or containerized environments, because they are built around Ubuntu),

there is a Python bug that affects fMRIPrep that drives the Linux kernel to kill processes as a response to

running out of memory.

Depending on the process killed by the kernel, fMRIPrep may crash with a ``BrokenProcessPool`` error

or hang indefinitely, depending on settings.

oesteban · 2019-05-03T18:59:15Z

docs/faq_tips_tricks.rst

+
+There is a Python bug that affects fMRIPrep when processes are killed for running out of memory. 
+While we are working on finding a solution that does not run up against this bug, this may take some time. 
+This can be most easily resolved by allocating more memory to the process, if possible. 


Please link here a bunch of discussions about the issue:

https://neurostars.org/t/memory-issue-when-processing-large-amount-of-data/2562

https://neurostars.org/t/how-much-ram-cpus-is-reasonable-to-run-pipelines-like-fmriprep/1086

https://neurostars.org/t/memory-allocation-issues-with-fmriprep-singularity-and-hpc/2759

https://neurostars.org/t/fmriprep-v1-0-12-hanging/1661

franklin-feingold · 2019-05-03T23:09:19Z

@oesteban Thank you for your review! Please find the latest commit applying your feedback

oesteban · 2019-05-05T01:47:21Z

Replaced by #1610 (all commits have been kept).

DOC: Add FAQ, Tips, Tricks section to RTD (cont. #1601)

franklin-feingold added 3 commits May 1, 2019 15:57

Merge pull request #2 from poldracklab/master

549be0e

update 5/1

add faq, tips, tricks section

49739b6

add faq, tips, tricks to index

7f928cc

franklin-feingold requested a review from oesteban May 1, 2019 23:52

fix circle error

f5e07ef

franklin-feingold requested a review from effigies May 2, 2019 17:08

emdupre reviewed May 2, 2019

View reviewed changes

newline each new sentence

4c77041

effigies reviewed May 2, 2019

View reviewed changes

franklin-feingold added 3 commits May 2, 2019 17:05

moved faq_tips_tricks

772a4fa

moved faq_tips_tricks to between outputs and contributors

Update faq_tips_tricks.rst

a91eaa3

update to render rst correctly

726a8fc

oesteban reviewed May 3, 2019

View reviewed changes

extended faq,tips, tricks

9fcc083

effigies added this to the 1.4.0 milestone May 4, 2019

oesteban mentioned this pull request May 5, 2019

DOC: Add FAQ, Tips, Tricks section to RTD (cont. #1601) #1610

Merged

oesteban closed this May 5, 2019

oesteban added a commit that referenced this pull request May 6, 2019

Merge pull request #1610 from oesteban/docs/md-wiki

423d777

DOC: Add FAQ, Tips, Tricks section to RTD (cont. #1601)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add FAQ, Tips, Tricks section to RTD #1601

Add FAQ, Tips, Tricks section to RTD #1601

franklin-feingold commented May 1, 2019

effigies commented May 2, 2019

emdupre May 2, 2019

franklin-feingold May 2, 2019

effigies May 2, 2019

effigies May 2, 2019

effigies May 2, 2019

effigies May 2, 2019

effigies May 2, 2019

effigies May 2, 2019

effigies May 2, 2019

effigies May 2, 2019

franklin-feingold commented May 3, 2019

oesteban left a comment

oesteban May 2, 2019

oesteban May 3, 2019

oesteban May 3, 2019

oesteban May 3, 2019

oesteban May 3, 2019

oesteban May 3, 2019

oesteban May 3, 2019

franklin-feingold commented May 3, 2019

oesteban commented May 5, 2019


		My ``FMRIPREP`` run is hanging...

		There is a Python bug that affects ``FMRIPREP`` when processes are killed for running out of memory. While we are working on finding a solution that does not run up against this bug, this may take some time. This can be most easily resolved by allocating more memory to the process, if possible. Additionally, consider using the ``--low-mem`` flag, which will make some memory optimizations at the cost of disk space in the working directory.

	FAQ, Tips, and Tricks
	=====================
	FAQ, Tips, and Tricks

	Should I run quality control of my images before running ``FMRIPREP``?
	Should I run quality control of my images before running fMRIPrep?
	------------------------------------------------------------------

		When using publicly available datasets, an additional concern is that images may have gone through some kind of preprocessing (see next question).


		What if I find some images have undergone some pre-processing already (e.g. my T1w image is already skull-stripped)?

	What if I find some images have undergone some pre-processing already (e.g. my T1w image is already skull-stripped)?
	What if I find some images have undergone some pre-processing already (e.g., my T1w image is already skull-stripped)?
	----------------------------------------------------

	Hence, supporting such images was considered very low priority for ``FMRIPREP``.
	Hence, supporting such images was considered very low priority for fMRIPrep.

		So for OpenFMRI, we've been excluding these subjects, and for user-supplied data, we would recommend reverting to the original, defaced, T1w images to ensure more uniform preprocessing.


		My ``FMRIPREP`` run is hanging...

	My ``FMRIPREP`` run is hanging...
	My fMRIPrep run is hanging...
	-----------------------------

	There is a Python bug that affects ``FMRIPREP`` when processes are killed for running out of memory.
	There is a Python bug that affects fMRIPrep when processes are killed for running out of memory.

	Yes. You should as well before any other processing/analysis.
	Yes. You should do so before any processing/analysis takes place.

-Some other times, QC checks will flag some images that we should carefully track throughout processing.
+Some exclusion criteria for data quality should be pre-specified before QC and any screening
+of the original data.
+Those exclusion criteria must be designed in agreement with the goals and challenges of the
+experimental design.
+For instance, when it is planned to run some cortical thickness analysis, images should be excluded
+even when they present the most subtle ghosts or other artifacts that may introduce biases in surface
+reconstruction.
+However, if the same artifactual data was planned to be used just as a reference for spatial
+normalization, some of those artifacts should be noted, but may not grant exclusion of the data.



		My fMRIPrep run is hanging...
		---------------------------------

-There is a Python bug that affects fMRIPrep when processes are killed for running out of memory.
+When running on Linux platforms (or containerized environments, because they are built around Ubuntu),
+there is a Python bug that affects fMRIPrep that drives the Linux kernel to kill processes as a response to
+running out of memory.
+Depending on the process killed by the kernel, fMRIPrep may crash with a ``BrokenProcessPool`` error
+or hang indefinitely, depending on settings.

Add FAQ, Tips, Tricks section to RTD #1601

Add FAQ, Tips, Tricks section to RTD #1601

Conversation

franklin-feingold commented May 1, 2019

Changes proposed in this pull request

Documentation that should be reviewed

effigies commented May 2, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

franklin-feingold commented May 3, 2019

oesteban left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

franklin-feingold commented May 3, 2019

oesteban commented May 5, 2019