Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support independent jobs across sessions #1175

Closed
chrisgorgo opened this issue Jun 21, 2018 · 5 comments · Fixed by #2036
Closed

Support independent jobs across sessions #1175

chrisgorgo opened this issue Jun 21, 2018 · 5 comments · Fixed by #2036
Labels
effort: medium Estimated medium effort task feature impact: medium Estimated medium impact task
Milestone

Comments

@chrisgorgo
Copy link
Contributor

It's certain that we will see more and more datasets like MyConnectome and Midnight Scan Club. To efficiently process such dataset we need to support submitting independent jobs across sessions. In practice this would mean:

@effigies
Copy link
Member

effigies commented Jul 2, 2018

I think one of the relevant problems to consider here is separate working directories, which will attempt to re-run anatomical. I think we need to ensure that all necessary anatomical inputs to the BOLD workflow are in the anatomical derivatives, and use those, rather than reaching into the working directory.

I think at one point I had an open issue about a DataGrabber for fmriprep derivatives. That could be a useful component here.

@oesteban
Copy link
Member

oesteban commented Jul 2, 2018

I think we need to ensure that all necessary anatomical inputs to the BOLD workflow are in the anatomical derivatives, and use those, rather than reaching into the working directory.

This is the key point here. +1

@poldrack
Copy link

poldrack commented Sep 1, 2018

just wanted to upvote this one again (just trying to analyze myconnectome data again).

@effigies effigies added this to To do in pipelines via automation Mar 12, 2019
@oesteban oesteban added impact: medium Estimated medium impact task effort: medium Estimated medium effort task labels Mar 16, 2019
@franklin-feingold franklin-feingold moved this from To do to Issues B-list in pipelines May 6, 2019
@oesteban oesteban added this to the 2.0.0 milestone Jul 11, 2019
@oesteban oesteban moved this from Issues B-list to Issues A-list in pipelines Jul 24, 2019
@oesteban oesteban moved this from Issues A-list to Development backlog in pipelines Jul 24, 2019
@oesteban oesteban moved this from Development backlog to Next week's targets in pipelines Sep 11, 2019
@oesteban oesteban moved this from Next week's targets to In progress in pipelines Sep 13, 2019
@carolinarsm
Copy link

Also bumping this much needed feature, so +1.

@oesteban
Copy link
Member

This will be possible when nipreps/smriprep#107 is merged in. Can't provide an ETA, but I'd like to see this fixed within 2 months.

oesteban added a commit to oesteban/fmriprep that referenced this issue Mar 14, 2020
… given

Summary
-------
This PR utilizes nipreps/smriprep#107 to allow a fast skip of the
anatomical workflow evaluation, and the need of maintaining the work
directory for it.

Closes nipreps#1175

Changes suggested
-----------------

  * [x] Added an ``--anat-derivatives`` flag to the command line.
  * [x] Integrated the new flag with the new config module.
  * [x] Estimate existing derivatives using *sMRIPrep*'s tooling.
  * [x] Pass in the existing derivatives and nipreps/smriprep#107
    skips the anatomical workflow fulfilling the workflow API.

Potential problems
------------------
This PR does not contain any measures to preserve the reproducibility of
*fMRIPrep* if the fast-track is used.
oesteban added a commit to oesteban/fmriprep that referenced this issue Mar 14, 2020
… given

Summary
-------
This PR utilizes nipreps/smriprep#107 to allow a fast skip of the
anatomical workflow evaluation, and the need of maintaining the work
directory for it.

Closes nipreps#1175

Changes suggested
-----------------

  * [x] Added an ``--anat-derivatives`` flag to the command line.
  * [x] Integrated the new flag with the new config module.
  * [x] Estimate existing derivatives using *sMRIPrep*'s tooling.
  * [x] Pass in the existing derivatives and nipreps/smriprep#107
    skips the anatomical workflow fulfilling the workflow API.

Potential problems
------------------
This PR does not contain any measures to preserve the reproducibility of
*fMRIPrep* if the fast-track is used.
oesteban added a commit to oesteban/fmriprep that referenced this issue Mar 18, 2020
… given

Summary
-------
This PR utilizes nipreps/smriprep#107 to allow a fast skip of the
anatomical workflow evaluation, and the need of maintaining the work
directory for it.

Closes nipreps#1175

Changes suggested
-----------------

  * [x] Added an ``--anat-derivatives`` flag to the command line.
  * [x] Integrated the new flag with the new config module.
  * [x] Estimate existing derivatives using *sMRIPrep*'s tooling.
  * [x] Pass in the existing derivatives and nipreps/smriprep#107
    skips the anatomical workflow fulfilling the workflow API.

Potential problems
------------------
This PR does not contain any measures to preserve the reproducibility of
*fMRIPrep* if the fast-track is used.
oesteban added a commit to oesteban/fmriprep that referenced this issue Mar 30, 2020
… given

Summary
-------
This PR utilizes nipreps/smriprep#107 to allow a fast skip of the
anatomical workflow evaluation, and the need of maintaining the work
directory for it.

Closes nipreps#1175

Changes suggested
-----------------

  * [x] Added an ``--anat-derivatives`` flag to the command line.
  * [x] Integrated the new flag with the new config module.
  * [x] Estimate existing derivatives using *sMRIPrep*'s tooling.
  * [x] Pass in the existing derivatives and nipreps/smriprep#107
    skips the anatomical workflow fulfilling the workflow API.

Potential problems
------------------
This PR does not contain any measures to preserve the reproducibility of
*fMRIPrep* if the fast-track is used.
@oesteban oesteban moved this from In progress to Awaiting review in pipelines Apr 2, 2020
oesteban added a commit to oesteban/smriprep that referenced this issue Apr 15, 2020
Postponing this relabeling created a deviation of the derivative written
out to the hard disk and the actual file passed on by the outputnode.

This PR should serve as a basis for the elimination of FSL FAST in
``recon-all`` -based workflows (cc/ @mgxd):

  - FAST and the new tailgating node should be moved inside the if
    branch for ``--fs-no-reconall``
  - The LUT mapping function should be reused to remap FS's ``aseg``
    into a 3-tissue brain segmentation.

Resolves: nipreps#176
Related: nipreps#170
Related: nipreps/fmriprep#1175
oesteban added a commit to oesteban/smriprep that referenced this issue Apr 15, 2020
Postponing this relabeling created a deviation of the derivative written
out to the hard disk and the actual file passed on by the outputnode.

This PR should serve as a basis for the elimination of FSL FAST in
``recon-all`` -based workflows (cc/ @mgxd):

  - FAST and the new tailgating node should be moved inside the if
    branch for ``--fs-no-reconall``
  - The LUT mapping function should be reused to remap FS's ``aseg``
    into a 3-tissue brain segmentation.

Resolves: nipreps#176
Related: nipreps#170
Related: nipreps/fmriprep#1175
oesteban added a commit to oesteban/smriprep that referenced this issue Apr 15, 2020
Postponing this relabeling created a deviation of the derivative written
out to the hard disk and the actual file passed on by the outputnode.

This PR should serve as a basis for the elimination of FSL FAST in
``recon-all`` -based workflows (cc/ @mgxd):

  - FAST and the new tailgating node should be moved inside the if
    branch for ``--fs-no-reconall``
  - The LUT mapping function should be reused to remap FS's ``aseg``
    into a 3-tissue brain segmentation.

Resolves: nipreps#176
Related: nipreps#170
Related: nipreps/fmriprep#1175
oesteban added a commit to oesteban/smriprep that referenced this issue Apr 16, 2020
Postponing this relabeling created a deviation of the derivative written
out to the hard disk and the actual file passed on by the outputnode.

This PR should serve as a basis for the elimination of FSL FAST in
``recon-all`` -based workflows (cc/ @mgxd):

  - FAST and the new tailgating node should be moved inside the if
    branch for ``--fs-no-reconall``
  - The LUT mapping function should be reused to remap FS's ``aseg``
    into a 3-tissue brain segmentation.

Resolves: nipreps#176
Related: nipreps#170
Related: nipreps/fmriprep#1175
oesteban added a commit to oesteban/smriprep that referenced this issue Apr 16, 2020
Postponing this relabeling created a deviation of the derivative written
out to the hard disk and the actual file passed on by the outputnode.

This PR should serve as a basis for the elimination of FSL FAST in
``recon-all`` -based workflows (cc/ @mgxd):

  - FAST and the new tailgating node should be moved inside the if
    branch for ``--fs-no-reconall``
  - The LUT mapping function should be reused to remap FS's ``aseg``
    into a 3-tissue brain segmentation.

Resolves: nipreps#176
Related: nipreps#170
Related: nipreps/fmriprep#1175
oesteban added a commit to oesteban/fmriprep that referenced this issue Apr 17, 2020
… given

Summary
-------
This PR utilizes nipreps/smriprep#107 to allow a fast skip of the
anatomical workflow evaluation, and the need of maintaining the work
directory for it.

Closes nipreps#1175

Changes suggested
-----------------

  * [x] Added an ``--anat-derivatives`` flag to the command line.
  * [x] Integrated the new flag with the new config module.
  * [x] Estimate existing derivatives using *sMRIPrep*'s tooling.
  * [x] Pass in the existing derivatives and nipreps/smriprep#107
    skips the anatomical workflow fulfilling the workflow API.

Potential problems
------------------
This PR does not contain any measures to preserve the reproducibility of
*fMRIPrep* if the fast-track is used.
oesteban added a commit to oesteban/fmriprep that referenced this issue Apr 17, 2020
… given

Summary
-------
This PR utilizes nipreps/smriprep#107 to allow a fast skip of the
anatomical workflow evaluation, and the need of maintaining the work
directory for it.

Closes nipreps#1175

Changes suggested
-----------------

  * [x] Added an ``--anat-derivatives`` flag to the command line.
  * [x] Integrated the new flag with the new config module.
  * [x] Estimate existing derivatives using *sMRIPrep*'s tooling.
  * [x] Pass in the existing derivatives and nipreps/smriprep#107
    skips the anatomical workflow fulfilling the workflow API.

Potential problems
------------------
This PR does not contain any measures to preserve the reproducibility of
*fMRIPrep* if the fast-track is used.
pipelines automation moved this from Awaiting review to Done Apr 28, 2020
@oesteban oesteban removed this from Done in pipelines Sep 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
effort: medium Estimated medium effort task feature impact: medium Estimated medium impact task
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants