ENH: Config module to replace long list of downstreamed arguments

[oesteban]

Includes addressing #2019, although I think that one could be addressed separately to simplify this already huge PR.

Requires: nipreps/niworkflows#475

Summary

This module implements the memory structures to keep a consistent, singleton config.

Example

>>> from fmriprep import config
>>> config_file = config.execution.work_dir / '.fmriprep.toml'
>>> config.to_filename(config_file)
>>> # Call build_workflow(config_file, retval) in a subprocess
>>> with Manager() as mgr:
>>>     from .workflow import build_workflow
>>>     retval = mgr.dict()
>>>     p = Process(target=build_workflow, args=(str(config_file), retval))
>>>     p.start()
>>>     p.join()
>>> config.load(config_file)
>>> # Access configs from any code section as:
>>> value = config.section.setting

The module also has a :py:func:to_filename function to allow writting out
the settings to hard disk in ToML format, which looks like:

[nipype]
crashfile_format = "txt"
get_linked_libs = false
nprocs = 8
omp_nthreads = 8
plugin = "MultiProc"
resource_monitor = false
stop_on_first_crash = false
version = "1.5.0-dev+gdffcb7815"

[execution]
bids_dir = "/data/openfmri/ds000005"
boilerplate_only = false
exec_env = "posix"
fs_license_file = "/opt/freesurfer/license.txt"
fs_subjects_dir = "/data/openfmri/ds000005/derivatives/freesurfer-6.0.1"
log_dir = "/data/openfmri/ds000005/derivatives/fmriprep/logs"
log_level = 15
low_mem = false
md_only_boilerplate = false
notrack = true
output_dir = "/data/openfmri/ds000005/derivatives"
reports_only = false
run_uuid = "20200302-174345_9ba9f304-82de-4538-8c3a-570c5f5d8f2f"
participant_label = [ "01",]
version = "20.0.1+13.g3ca42930.dirty"
work_dir = "/home/oesteban/tmp/fmriprep-ds005/fprep-work2"
write_graph = false

[workflow]
anat_only = false
aroma_err_on_warn = false
aroma_melodic_dim = -200
bold2t1w_dof = 6
cifti_output = false
fmap_bspline = false
force_syn = false
hires = true
ignore = []
internal_spaces = "MNI152NLin2009cAsym"
longitudinal = false
medial_surface_nan = false
run_reconall = true
skull_strip_fixed_seed = false
skull_strip_template = "OASIS30ANTs"
t2s_coreg = false
use_aroma = false

[nipype.plugin_args]
maxtasksperchild = 1
raise_insufficient = false

This config file is used to pass the settings across processes,
using the :py:func:load function.

Other responsibilities of the config module:

• Switching Python's multiprocessing to forkserver mode.

• Set up new logger levels (25: IMPORTANT, and 15: VERBOSE).

• Set up a warnings filter as soon as possible.

• Initialize runtime descriptive settings (e.g., default FreeSurfer license,
  execution environment, nipype and fMRIPrep versions, etc.).

• Automated I/O magic operations:

  • :obj:Path <-> :obj:str <-> :obj:Path).
  • :py:class:~niworkflows.util.spaces.SpatialReferences <-> :obj:str <->
    :py:class:~niworkflows.util.spaces.SpatialReferences
  • :py:class:~bids.layout.BIDSLayout <-> :obj:str <->
    :py:class:~bids.layout.BIDSLayout

Potential utilizations of the config module

• Crawling versions of relevant software and storing them in the read-only section (environment) of the config file, so that those are then reported in the log folder and within sentry.
• Keeping track of random seeds (generating them and keeping them for future execution).

[effigies]

@oesteban Here's the merge conflict between this and 20.0.2:

def main():
    """Entry point."""
    import os
    import sys
    import gc
    from nipype import config as ncfg
    from multiprocessing import Process, Manager
    from .parser import parse_args
    from ..utils.bids import write_derivative_description

    parse_args()

    sentry_sdk = None
    if not config.execution.notrack:
        import sentry_sdk
        from ..utils.sentry import sentry_setup
        sentry_setup()

    config_file = config.execution.work_dir / '.fmriprep.toml'
    config.to_filename(config_file)

<<<<<<< HEAD
    # Precedence: --fs-license-file, $FS_LICENSE, default_license
    license_file = opts.fs_license_file

    if license_file is None and os.getenv("FS_LICENSE"):
        license_file = Path(os.getenv("FS_LICENSE"))

    if license_file is None and os.getenv("FREESURFER_HOME"):
        license_file = Path(os.getenv("FREESURFER_HOME")) / "license.txt"

    # FreeSurfer license
    if license_file is None or not license_file.exists():
        raise RuntimeError("""\
ERROR: a valid license file is required for FreeSurfer to run. fMRIPrep looked for an existing \
license file at several paths, in this order: 1) command line argument ``--fs-license-file``; \
2) ``$FS_LICENSE`` environment variable; and 3) the ``$FREESURFER_HOME/license.txt`` path. Get it \
(for free) by registering at https://surfer.nmr.mgh.harvard.edu/registration.html""")
    os.environ["FS_LICENSE"] = str(license_file.absolute())

    # Retrieve logging level
    log_level = int(max(25 - 5 * opts.verbose_count, logging.DEBUG))
    # Set logging
    logger.setLevel(log_level)
    logger.addHandler(logging.StreamHandler())
    nlogging.getLogger('nipype.workflow').setLevel(log_level)
    nlogging.getLogger('nipype.interface').setLevel(log_level)
    nlogging.getLogger('nipype.utils').setLevel(log_level)

    # Call build_workflow(opts, retval)
=======
    # Call build_workflow(config_file, retval) in a subprocess
>>>>>>> oesteban/maint/config-file
    with Manager() as mgr:
        from .workflow import build_workflow
        retval = mgr.dict()
        p = Process(target=build_workflow, args=(str(config_file), retval))
        p.start()
        p.join()

        retcode = p.exitcode or retval.get('return_code', 0)
        fmriprep_wf = retval.get('workflow', None)

    config.load(config_file)

Would you prefer to resolve that in this PR, or after this is done? (Trying to decide whether to merge the latest tag into master.)

[oesteban]

Issue found: Unused variable 'SpatialReferences'

[effigies]

Should I assume if Codacy tagged this, you'll take care of it?

(BTW Codacy comments are completely spamming my feed. Is there any way to have it make just one review?)

[oesteban]

Yes, I have the commit done, was just waiting for Circle to finish so that the build didn't get cancelled.

I have changed Codacy's settings so that only one summary comment will be posted.

[oesteban]

Issue found: Consider possible security implications associated with check_call module.

[oesteban]

when not specifying --output-spaces.

Also, specifying --output-spaces without any arguments should not produce outputs, but it's currently not possible.

Good catch. I've addressed both use cases and added a test that doesn't cover all possibilities but is pretty close.

I've also tested this on ls5 on 15 subjects from the ABCD Study and worked like a charm.

I'll hold off until I get more feedback (cc/ @mgxd, @effigies and perhaps also interesting to check for @emdupre, @jdkent, @mattcieslak).

[effigies]

This doesn't really seem like a mock, in that it's not resetting values when finished. Is the goal to eventually move to a mock?

[oesteban]

Yes, it's going to be tricky, but that would be the right thing. For now, since this is only utilized in generating documentation, it seems to work well enough.

[effigies]

Should I assume if Codacy tagged this, you'll take care of it?

(BTW Codacy comments are completely spamming my feed. Is there any way to have it make just one review?)

[effigies]

Why is this in finally?

[oesteban]

Theoretical reason: it is explicit those are to be run regardless the branch flow went through.
Practical reason: so that my linter does not complain about imports after plain code.

Happy to take out from the finally if that feels more appropriate.

[oesteban]

I was thinking @josephmje may want to weigh in this one too.

[oesteban]

I think this is good to go. Happy to send a quick PR to address #2018 (comment) if it will make code clearer.

[josephmje]

I was getting an error on this line when running this. It looks like config.nipype.omp_nthread is missing an 's'.

[effigies]

Good catch. Care to submit a quick fix?

[oesteban]

Good catch. Care to submit a quick fix?

Yay :) - please f-strings :)