Add Flux executor to nextflow (#3412)

this work adds the Flux Framework executor, meaning
a Nextflow user can submit jobs to flux. This addition
includes the executor, tests, a documentation tutorial,
addition to the executor.rst page, and a Docker and/or
DevContainers environment for the interested developer.

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

.gitignore

@@ -6,6 +6,7 @@
 .cache/*
 .nextflow*
 .node-nextflow*
+.devcontainer
 **/build/**
 build/**
 modules/**/build/

README.md

@@ -105,6 +105,7 @@ Currently the following clusters are supported:
   + [LSF](https://www.nextflow.io/docs/latest/executor.html#lsf)
   + [SLURM](https://www.nextflow.io/docs/latest/executor.html#slurm)
   + [PBS/Torque](https://www.nextflow.io/docs/latest/executor.html#pbs-torque)
+  + [Flux](https://www.nextflow.io/docs/latest/executor.html#flux)
   + [HyperQueue (beta)](https://www.nextflow.io/docs/latest/executor.html#hyperqueue)
   + [HTCondor (beta)](https://www.nextflow.io/docs/latest/executor.html#htcondor)
   + [Moab](https://www.nextflow.io/docs/latest/executor.html#moab)

docker/flux/.devcontainer/Dockerfile

@@ -0,0 +1,20 @@
+FROM fluxrm/flux-sched:focal
+
+# From project root!
+# docker build -f docker/flux/.devcontainer/Dockerfile -t nextflow-flux .
+# docker run -it nextflow-flux
+# make
+
+LABEL maintainer="Vanessasaurus <@vsoch>"
+
+# Pip not provided in this version
+USER root
+RUN apt-get update && apt-get install -y python3-venv default-jdk
+ENV PYTHONPATH=/usr/lib/flux/python3.8
+
+# For optional usage of Python bindings.
+RUN python3 -m pip install IPython
+ENV PATH=/env/bin:${PATH}
+WORKDIR /code
+COPY . /code
+ENTRYPOINT /bin/bash

docker/flux/.devcontainer/devcontainer.json

@@ -0,0 +1,33 @@
+{
+    // Python is provided as a courtesy (it comes with base container)
+    "name": "Nextflow with Flux Executor Python 3.6",
+
+    // This assumes copying this directory to the root of Nextflow
+    "dockerFile": "Dockerfile",
+    "context": ".",
+
+    "customizations": {
+      "vscode": {
+        "settings": {
+          "terminal.integrated.defaultProfile.linux": "bash",
+
+          // Ensure that Python autocomplete works out of the box
+          "python.autoComplete.extraPaths": [
+            "/usr/lib/flux/python3.8",
+            "/usr/lib/python3.8/site-packages",
+          ],
+          "python.analysis.extraPaths": [
+            "/usr/lib/flux/python3.8",
+            "/usr/lib/python3.8/site-packages",
+          ]
+        },
+        // Note to Flux Developers! We can add extensions here that you like
+        "extensions": [
+			"ms-python.python",
+			"vscjava.vscode-java-pack"
+		],
+      },
+    },
+    // Needed for git security feature
+    "postStartCommand": "git config --global --add safe.directory /workspaces/nextflow"
+  }

docker/flux/README.md

@@ -0,0 +1,8 @@
+# Flux Container Environment
+
+In this directory we include a [Dockerfile](.devcontainer/Dockerfile)
+along with a [VSCode Developer Container](https://code.visualstudio.com/docs/devcontainers/containers) 
+environment that you can put at the root of the project to be provided with a Flux agent and the dependencies
+needed to build Nextflow. See the [documentation](https://www.nextflow.io/docs/latest/flux.html) 
+for the latest for how to use them.
+

docs/basic.rst

@@ -96,6 +96,7 @@ The following batch schedulers are supported:
 * `Univa grid engine <http://www.univa.com/>`_
 * `Platform LSF <http://www.ibm.com/systems/technicalcomputing/platformcomputing/products/lsf/>`_
 * `Linux SLURM <https://computing.llnl.gov/linux/slurm/>`_
+* `Flux Framework <https://flux-framework.org/>`_
 * `PBS Works <http://www.pbsworks.com/gridengine/>`_
 * `Torque <http://www.adaptivecomputing.com/products/open-source/torque/>`_
 * `HTCondor <https://research.cs.wisc.edu/htcondor/>`_

docs/executor.rst

@@ -85,6 +85,27 @@ Resource requests and other job characteristics can be controlled via the follow
 * :ref:`process-time`
 
 
+.. _flux-executor:
+
+Flux Framework Executor
+=======================
+
+The ``flux`` executor allows you to run your pipeline script by using `Flux Framework <https://flux-framework.org>`_.
+
+Akin to slurm, Nextflow manages each process as a separate job that is submitted to the cluster by using the ``flux mini submit`` command.
+
+To enable the Flux executor simply set ``process.executor = 'flux'`` in the ``nextflow.config`` file.
+
+Resource requests and other job characteristics can be controlled via the following process directives:
+
+* :ref:`process-clusterOptions`
+* :ref:`process-cpus`
+* :ref:`process-queue`
+* :ref:`process-time`
+
+.. note:: Flux does not support specifying memory. 
+
+
 .. _ga4ghtes-executor:
 
 GA4GH TES

docs/flux.rst

@@ -0,0 +1,213 @@
+.. _flux-page:
+
+**************
+Flux Framework
+**************
+
+`Flux Framework <https://flux-framework.org/>`_ "flux" is modern resource manager that can span the space between
+cloud and HPC. If your center does not provide Flux for you, you can `build Flux on your own <https://flux-framework.readthedocs.io/en/latest/quickstart.html#building-the-code>`_
+and launch it as a job under your resource manager of choice (e.g., SLURM or on the cloud.)
+
+
+Tutorial
+========
+
+In the `docker/flux <https://github.com/nextflow-io/nextflow/tree/master/docker/flux>`_ directory we provide 
+a `Dockerfile for interacting with Flux <https://github.com/nextflow-io/nextflow/tree/master/docker/flux/.devcontainer/Dockerfile>`_
+
+along with a `VSCode Developer Container <https://code.visualstudio.com/docs/devcontainers/containers>`_
+environment that you can put at the root of the project to be provided with a Flux agent and the dependencies
+needed to build Nextflow. There are two ways to use this:
+
+ - Build a container from scratch and bind your code to it (e.g., for development or testing)
+ - Use VSCode and DevContainers to create a more seamless environment.
+
+Both strategies are described below. For this tutorial, you will generally want 
+to prepare a workflow to use the ``flux`` executor, create an environment with Flux, start a Flux instance, and
+interact with it.
+
+Prepare Nextflow Workflow 
+-------------------------
+
+For your workflow, to use Flux you'll want to specify it in your config. Here is an example 
+``nextflow.config``
+
+
+.. code-block:: console
+
+    // This is an example config for running with flux.
+
+    manifest {
+      mainScript = 'demo.nf'
+      homePage = 'https://github.com/nextflow-io/nextflow/tree/master/docker/flux'
+      description = 'Demo using Nextflow with Flux'
+    } 
+
+    process {
+      executor = 'flux'
+    }
+
+Note that the executor is ``flux``. For additional settings for Flux, see :ref:`flux-executor`.
+Here is an a demo workflow ``demo.nf`` of a job we want to run!
+
+.. code-block:: console
+
+    breakfast = Channel.from '🥞️', '🥑️', '🥧️', '🍵️', '🍞️'
+ 
+    workflow {
+      haveMeal(breakfast)
+    }
+    
+    process haveMeal {
+      debug true
+      input:
+      val food
+      """
+      printf '$food for breakfast!'
+      """
+    }
+
+We will be using these files to run our test workflow. Next, assuming you don't have one handy,
+let's set up an envrionment with Flux.
+
+Container Environment
+---------------------
+
+You can choose just to build the Docker image from the root of the repository:
+
+.. code-block:: console
+    
+    $ docker build -f docker/flux/.devcontainer/Dockerfile -t nextflow-flux .
+
+
+And then shell into the container for a development environment. You'd need to bind
+the present working directory to ``/code`` to see your local changes in the container:
+
+.. code-block:: console
+
+    $ docker run -it -v $PWD:/code nextflow-flux 
+
+
+You can also move the .devcontainer directory to the root of your repository, and just
+open it in VSCode.
+
+.. code-block:: console
+
+    $ cp -R docker/flux/.devcontainer .devcontainer
+
+
+Then open in vscode, and select to "Re-open in container"
+
+.. code-block:: console
+
+    $ code .
+
+Then you should be able to open a terminal (Terminal -> New Terminal)
+to interact with the command line. Try running `make`` again!
+Whichever of these two approaches you take, you should be in a container 
+environment with ``flux`` on the path.
+
+Start a Flux Instance 
+---------------------
+
+Once in your container, you can start an interactive flux instance (from which you can submit jobs on
+the command line to test with Nextflow) as follows:
+
+.. code-block:: console
+
+    $ flux start --test-size=4
+
+
+Getting Familiar with Flux
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. note::
+
+    This step is optional!
+
+Here is an example of submitting a job and getting the log for it.
+
+.. code-block:: console
+
+    $ flux mini submit echo "HELLO MOTO"
+    ƒEzWqspb
+
+
+And then getting the log for it:
+
+
+.. code-block:: console
+
+    $ flux job attach ƒEzWqspb
+    HELLO MOTO
+
+
+Try submitting a longer job:
+
+.. code-block:: console
+
+    $ flux mini submit sleep 60
+
+
+And then seeing it in the jobs listing.
+
+.. code-block:: console
+
+    $ flux jobs
+           JOBID USER     NAME       ST NTASKS NNODES     TIME INFO
+       ƒ4tkMUAAT root     sleep       R      1      1   2.546s ab6634a491bb
+
+
+We are going to be issuing similar submit commands via Nextflow!
+
+Submitting with Nextflow
+------------------------
+
+Prepare your ``nextflow.config`` and ``demo.nf`` in the same directory.
+
+.. code-block:: console
+
+    $ ls .
+    demo.nf    nextflow.config 
+
+
+If you've installed Nextflow already with the flux executor, you are good to go! If you are working 
+with development code and need to build nextflow: 
+
+.. code-block:: console
+
+    $ make assemble
+
+Make sure ``nextflow`` is on your PATH (here we are in the root of Nextflow):
+
+.. code-block:: console
+
+    $ export PATH=$PWD:$PATH
+    $ which nextflow
+    /workspaces/nextflow/nextflow
+
+Then cd into the directory with your config and demo file:
+
+.. code-block:: console
+
+    $ cd docker/flux
+
+
+And then run the workflow with flux!
+
+.. code-block:: console
+
+    $ nextflow -c nextflow.config run demo.nf
+
+    # nextflow -c nextflow.config run demo.nf 
+    N E X T F L O W  ~  version 22.10.0
+    Launching `demo.nf` [clever_blackwell] DSL2 - revision: f8cda838cb
+    executor >  flux (5)
+    [4c/f162db] process > haveMeal (3) [100%] 5 of 5 ✔
+    🥞️ for breakfast!
+    🍞️ for breakfast!
+    🍵️ for breakfast!
+    🥑️ for breakfast!
+    🥧️ for breakfast!
+
+And that's it! You've just run a workflow using nextflow and Flux. 

modules/nextflow/src/main/groovy/nextflow/executor/ExecutorFactory.groovy

@@ -45,6 +45,7 @@ class ExecutorFactory {
     final static Map<String, Class<? extends Executor>> BUILT_IN_EXECUTORS = [
             'nope': NopeExecutor,
             'local': LocalExecutor,
+            'flux': FluxExecutor,
             'sge':  SgeExecutor,
             'oge':  SgeExecutor,
             'uge':  SgeExecutor,