Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Postpone ``pandoc`` execution to the end of all processing #1709

Closed
oesteban opened this issue Jul 20, 2019 · 0 comments

Comments

@oesteban
Copy link
Contributor

commented Jul 20, 2019

Soichi found that the memory peak at the beginning of fMRIPrep is caused by pandoc, which is aligned to this open issue in pandoc.

Since that issue has been open for a while now, but we are not going to think of an alternative just for the conversion of markdown to html to generate the boilerplate, a good solution is to postpone pandoc to be the latest process run, and also offer a flag to totally skip the conversion.

@oesteban oesteban self-assigned this Jul 20, 2019

@oesteban oesteban added this to To do in pipelines via automation Jul 20, 2019

oesteban added a commit to oesteban/fmriprep that referenced this issue Jul 21, 2019

FIX: Postpone ``pandoc`` conversion of boilerplate after workflow has…
… fully run.

This PR addresses the memory peak caused by pandoc, running the process
at the very end of fMRIPrep.

We probably want to explicitly clear up as many objects as we can from
memory and force gc before the subprocess opens pandoc, but that's left
for a future PR.

This PR just closes poldracklab#1709 with the delayed pandoc run. I'm not
implementing the flag to avoid the boilerplate generation because I'm
not sure we want to introduce an additioinal argument for this.

Coincidentally, it also closes poldracklab#1667 - now the deletion of citation
files is done inside a try..catch where ``FileNotFoundError`` is
dismissed. That should avert the issue, while keeping the functionality
of poldracklab#1567.

@effigies effigies moved this from To do to In progress in pipelines Jul 22, 2019

pipelines automation moved this from In progress to Done Jul 22, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
1 participant
You can’t perform that action at this time.