Airflow Imaging plugins

Airflow plugins providing support for preprocessing of neuroimaging data.

The following packages are provided:

• airflow_freespace: Sensors that check the amount of free space on the disk and waits until enough free space is available
• airflow_pipeline: Operators and helpers to build generic processing pipelines
• airflow_scan_folder: Operators used to scan folders for new work
• airflow_spm: Operators adapting Matlab and SPM 12 to work inside Airflow

Usage

Imaging data is organised by folders, where each fist-level folder represents a scanning session.

A 'pipeline' represents the steps to process a folder containing one scanning session. In Airflow, we use the XCOM mechanism to transmit data from one step of the pipeline to the next step. This is why each processing pipelines need to start with airflow_pipeline.operators.PreparePipelineOperator as it injects into XCOM the necessary information that is required for the other *PipelineOperator:

All Python callback functions provided to those operators can use as arguments the following variables coming from XCOM:

• folder
• session_id
• participant_id (optional)
• scan_date (optional)
• output
• error
• dataset
• matlab_version
• spm_version
• spm_revision
• provenance_details
• provenance_previous_step_id
• relative_context_path

See airflow_pipeline.pipelines.PIPELINE_XCOMS for an up-to-date list

List of plugins

• airflow_spm.operators.SpmOperator: Executes SPM or just Matlab
• airflow_spm.operators.SpmPipelineOperator: Executes a pipeline on SPM, where a 'pipeline' is a function implemented in SPM
• airflow_freespace.operators.FreeSpaceSensor: Waits for enough free disk space on the disk.
• airflow_scan_folder.operators.ScanFlatFolderOperator: Triggers a DAG run for a specified dag_id for each scan folder discovered in a folder.
• airflow_scan_folder.operators.ScanDailyFolderOperator: Triggers a DAG run for a specified dag_id for each scan folder discovered in a daily folder.
• airflow_scan_folder.operators.ScanFlatFolderPipelineOperator: Triggers a DAG run for a specified dag_id for each folder discovered in a parent folder, where the parent folder location is provided by the pipeline XCOMs.
• airflow_pipeline.operators.PreparePipelineOperator: An operator that prepares the pipeline
• airflow_pipeline.operators.PythonPipelineOperator: A PythonOperator that moves XCOM data used by the pipeline
• airflow_pipeline.operators.BashPipelineOperator: A BashOperator that registers provenance information in the pipeline
• airflow_pipeline.operators.DockerPipelineOperator: A DockerOperator that registers provenance information in the pipeline

Python version: 3

Installation

  pip3 install from git: pip3 install git+https://github.com/LREN-CHUV/airflow-imaging-plugins.git@master#egg=airflow_imaging_plugins

Setup and configuration

Airflow setup for MRI scans pipeline:

• In Airflow config file, add the [spm] section with the following entries:
  • SPM_DIR: root path to the installation of SPM

build

Run ./build.sh.

Push on PyPi

Run ./publish.sh.

(This builds the project prior to pushing it).

Acknowledgements

This work has been funded by the European Union Seventh Framework Program (FP7/2007­2013) under grant agreement no. 604102 (HBP)

This work is part of SP8 of the Human Brain Project (SGA1).