Running a Job

Writing a workflow

Before running a job from CWL, you first must write a workflow. Writing your own workflows is further described in the section writing your own workflows. As an example, we will be using the xbrain pipeline, which is already written. This example can be found under saber/xbrain/example. However, it is necessary to modify the job file, as it requires a BOSS token to be run. You can open the example_job.yml in your favorite editor and simply add your provided token.

Opening a shell into the container

It is first necessary to open a shell into the running container. You can view all running containers with docker ps. The container you want to access will be called saber_cwl_parser_1

> docker ps

CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS                            PORTS                                        NAMES
f476bf393335        saber_cwl_parser   "tail -F root"           5 seconds ago       Up 4 seconds                                                                   saber_cwl_parser_1
06f24a760d9d        saber_webserver    "/entrypoint.sh webs…"   6 seconds ago       Up 5 seconds (health: starting)   5555/tcp, 8793/tcp, 0.0.0.0:8080->8080/tcp   saber_webserver_1
62f97890412b        postgres:9.6               "docker-entrypoint.s…"   8 seconds ago       Up 6 seconds                      5432/tcp                                     saber_postgres_1
f69745ffaa58        datajoint/mysql            "docker-entrypoint.s…"   8 seconds ago       Up 6 seconds                      0.0.0.0:3306->3306/tcp, 33060/tcp            saber_datajoint_1

Then you can use the following command to open a shell in that container

docker exec -it saber_cwl_parser_1 /bin/bash 

This should take you to the root directory in the container. From there, you should navigate to where your workflow is stored. For our example,

cd saber/xbrain/example

Building a workflow

Building a workflow creates the custom SABER containers and pushes them to AWS.

The syntax for building a workflow

conduit build <workflow file> <job file> 

so, for our example, this would be

conduit build workflow.cwl example_job.yml

Parsing the CWL

Parsing the CWL will create an Airflow DAG. This is how the CWL will be executed.

The syntax for parsing a workflow is

conduit parse <workflow file> <job file> [--parameterize <sweep file>] [--build]

so, for our example, this would be

conduit parse workflow.cwl example_job.yml

The two optional arguments are parameterize and build. Parameterize sweeps over a list of parameters and runs a workflow with the cartesian product of the sets of parameter values defined in the file. The format for these files can be found in parameter sweeps and optimization. Build creates the containers as well as parsing the workflow.

Optimizing workflows

Optimizing a workflow is different than running a sweep, as sometimes it might be necessary to update which subset of paremeters is being swept based on the results of the last run. Luckily, SABER supports this.

The syntax for optimizing a workflow is

conduit optimize <workflow file> <job file> <sweep file> [--output <file>] [--sampling-strategy <strategy>] [--max-iterations <number>]

We will be using the 2D xbrain example for optimization, as it runs faster and can be run on a laptop without much problem. This example can be found under saber/xbrain/example2d. An example of an optimization command is below.

conduit optimize optimization2d.cwl job.yml params.yml --sampling-strategy=random --max-iterations=10 --output=results.txt

This would optimize the parameters defined in the sweep file with a random sampling strategy for 10 iterations and then save the output to a file called results.txt.

Currently, random is the only sampling strategy that we have included. However, we have a standardized API that allows the user to define their own sampling strategies. The details of this API can be found in conduit/utils/parameterization.py.

Job naming and workflow directory structure

SABER creates names and batch job definitions based on the directory structure of a job.

This structure is as follows:

job_name
  |-> job.yml (job definition file)
  |-> sweep.yml (parameter sweep file, optional)
  |-> workflow.cwl (workflow definition file)

This creates an Airflow DAG called workflow_job_name. If you change the job file or workflow file within this folder and re-run the script, the new DAG will overwrite the old one. If you wish to preserve that dag, simply copy and rename the directory. Running the cwl-to-dag script on the new cwl workflow and job in the new directory will place that DAG in the Airflow GUI, meaning you can trigger both at once.

Note: Due to the way Airflow database backend works, if you delete a dag or CWL from the filesystem, the DAG will persist in the Airflow GUI but be unclickable. This is a bug we intend to solve.

WARNING: Changing the names or amount of inputs to a job after you have built the dag will result in an error when trying to rebuild the dag. Please either rename the dag by renaming the folder, job, or cwl file.

Job notification

At this time, there is no way to be notified when a job is complete. In the future, we will support adding an optional email operator to the end of the Airflow DAG.