Example Batching

The runPipeline.sh script can also be wrapped by another driver script to allow for batching multiple datasets or different fields of view (FOV) to run consecutively. In this example there are multiple FOVs to be processed and want to be able to run the pipeline multiple times without user interaction. The FOVs are stored in separate folders each containing the sub directories described on the Image Data Preparation page. These folders have a capital F followed by 3 digits representing the FOV number; this number is also appended to the base file name of the input files. The following shows an example python script that reads a starting FOV, the number of FOVs to be processed, a base file name, and finally the parent directory. The main functionality of this script is formatting input parameters to be used by the run pipeline script.

#!/usr/bin/env python
import sys
import subprocess
import os

if __name__ == '__main__':
    
    if len(sys.argv) != 5:
        print(                 \
         "Error expecting: " + \
         "starting_fov "     + \
         "num_fovs "         + \
         "base_dir "         + \
         "base_filename "    + \
         "as parameters"       \
        )
        exit(1)
    # read command line parameters converting where necessary
    starting_fov  =             int(sys.argv[1])
    num_fovs      =             int(sys.argv[2])
    base_dir      = os.path.abspath(sys.argv[3])
    base_filename =                 sys.argv[4]
    # Loop over the specified FOVs
    for fov in range(starting_fov,starting_fov+num_fovs):
        # Create FOV string for directory and base name
        # e.g i fov=1 then fov_name = "F001"
        fov_name = f"F{fov:03d}"
        # parent directory of FOV containing the ExSeq directories 
        # (1_deconvolution, 2_color-correction, ...)
        output_dir = os.path.join(base_dir  , fov_name) 
        input_dir  = os.path.join(output_dir, "0_raw")
        log_dir    = os.path.join(output_dir, "logs")
        report_dir = os.path.join(log_dir   , "imgs")

        # In this case want to store and create a log and
        # reporting directory per FOV but it is fine
        # if they already exist
        os.makedirs(log_dir   , exist_ok=True)
        os.makedirs(report_dir, exist_ok=True)

        final_base = base_filename + "-" + fov_name

        # Create the exact string would run on command line
        # specifying the appropriate folders and importantly 
        # forcing yes to all interactive prompts
        commandStr = "./runPipeline.sh " + \
                     " -I " + input_dir  + \
                     " -O " + output_dir + \
                     " -b " + final_base + \
                     " -L " + log_dir    + \
                     " -i " + report_dir + \
                     " -y"

        # Run the pipeline waiting for return
        # and piping output through
        print("=========== Processing " + fov_name + " ===========")
        subprocess.run(                        \
          commandStr, shell=True, check=True,  \
          stderr=sys.stderr, stdout=sys.stdout \
        )