Alternative computing infrastructure

BioRels was built with the idea it can be run anywhere. The singularity container allows you to have all of the packages you'll need in one location, reducing the installation load to a minimum. However, we understand you might not have an SGE cluster at your disposal and want to use something else. Although we can't offer those different options, we can provide to you as much guidance as we can, and support. Please follow the steps below and do not hesitate to raise a ticket if you need help.

Environment variables

Please open setenv.sh located in $TG_DIR/BACKEND/SCRIPT/SHELL/setenv.sh. setenv.sh contains all the global environment variables that will be sourced (loaded) every time a BioRels script is run. Feel free to add and set any environment variables that are the same across all scripts. This can be the port used, the cluster name or adding any libraries to $LD_LIBRARY_PATH or executables to the $PATH

Push environment variables to the container

Although the environment variables are loaded each time a BioRels script is executed, they will not automatically transfer to the container. To ensure these variables are available within the container, open the file $TG_DIR/BACKEND/CONTAINER/env-file.txt. This script is triggered when the Singularity container runs and is responsible for transferring global environment variables into the container's environment. For example, it includes variables such as PATH, TG_DIR, and Postgres database connection details (e.g., PGPASSWORD, PGUSER, DB_PORT, DB_HOST, DB_NAME, etc.).

Similarly, any variables you define in setenv.sh must also be added to env-file.txt. For instance, if you have a variable named A, simply add the following line to env-file.txt: A=$A

Adding a new clustering tool to the BioRels configuration:

In CONFIG_USER, located in BACKEND/SCRIPT/CONFIG/CONFIG_USER, you will need to look for the global variable MONITOR_TYPE and define a new value for the clustering tool you will be using. A good guideline would be NAME_CLUSTER where NAME is the name of the clustering tool.

Now that we have told BioRels that there is a new computer clustering application, we need to define it's behavior. The good thing is, all the use cases are within 1 file, called loader_qengine.php.

Modifying loader_submission:

Please open BACKEND/SCRIPT/LIB/loader_submission.php. This specific script handles the submission and monitoring of the different BioRels scripts. It implements a few functions:

submit_biorels_job:

When a job meet all the requirements, submit_biorels_job will be triggered to actually submit the job to the queue. For that it will look at the type of cluster and the type of job. If it is a batch script (job array), it will call submit_batch function. Otherwise it will submit the job to the queue. In this specific function, you will need to modify the section about the memory: if ($JOB_INFO['MEM']!=-1) { if ($GLB_VAR['MONITOR_TYPE']=='SGE_CLUSTER') { $ADD_DESC.=' -l m_mem_free='.$JOB_INFO['MEM'].'M -l h_rss='.$JOB_INFO['MEM'].'M '; } /// You can add other cluster specific parameters here: // else if ($GLB_VAR['MONITOR_TYPE']=='YOUR_CLUSTERING_TOOL') // { // $ADD_DESC.=' '; // } }

This is a pretty rare situation in which your script would require a lot more memory than usual. If it is the case, you can add specific cluster submission parameters there.

Next, you will need to modify this section of the code: ` // else if ($GLB_VAR['MONITOR_TYPE']=='YOUR_CLUSTERING_TOOL') // { // /// YOUR CODE HERE // /// The goal here is to submit the job to your clustering tool // /// There are 2 situations. If the job is a single job, you can submit it directly // /// If the job is a batch job, you need to call the function submit_batch($JOB_ID,$JOB_INFO) // /// which will return the job id // /// Then you store the job id in the GLB_RUN_JOBS array // if ($JOB_INFO['RUNTIME']=='S') // { // //// SUBMIT HERE // /// The name of the job MUST be $GLB_VAR['JOB_PREFIX'].'_'.$JOB_ID // /// GET THE submitted job id as $SUBMITTED_JOB_ID // ///$GLB_RUN_JOBS[$SUBMITTED_JOB_ID]=$JOB_ID; // } // else // { // $job_array_pid=submit_batch($JOB_ID,$JOB_INFO); // $GLB_RUN_JOBS[$job_array_pid]=$JOB_ID; // }

// }

`

That will define the submission of single jobs to your cluster engine. Uncomment those lines and add your code to submit the job and return the corresponding job id. To be consistent, the name of the job during submission must use the JOB_PREFIX value defined by the user in CONFIG_USER, followed by underscore and the JOB_ID. As you can see, if it a batch script, it will call the submit_batch function, which we will see next.

submit_batch:

As in the name, this function will submit a job array/batch job. This is required when the dataset is massive enough that it requires parallelization. At the end of that specific function you can uncomment the following lines: else if ($GLB_VAR['MONITOR_TYPE']=='YOUR_CLUSTERING_TOOL') { /// execute your job array and retrieve the job id ($JOB_ARRAY_ID) // return $JOB_ARRAY_ID; }

And add your code in it to submit the job array and retrieve the job array identifier, then return it.

monitor_running_jobs:

Once submitted, we need to check that it is still running or it is done. Please uncomment the following section: /* else if ($GLB_VAR['MONITOR_TYPE']=='YOUR_CLUSTERING_TOOL') { /// YOUR CODE HERE /// The goal here is to get the list of running jobs and return it as an array $val /// so that in the next step we can compare the list of running jobs with the list of jobs we have submitted } */

Modify YOUR_CLUSTERING_TOOL by the corresponding value. Then, you will need to execute some bash commands to retrieve your jobs. You can look at how it is done for SGE_CLUSTER: exec('qstat | egrep "('.$GLB_VAR['JOB_PREFIX'].'_|arrayjob|NNPS)" ',$val); You can notice that we perform a qstat, which will list ALL jobs. We then perform a egrep based on the JOB_PREFIX to retrieve the corresponding BioRels jobs. It will then store the results in an array called $val. Please be mindful that in the next step, each record in $val will be broken down by spaces, and the ID will be defined as the first column in that record. If it is not the case for your process, please modify your code accordingly.

prepare_batch:

Last script is how are configured job array/batch script. In BioRels, prepare_batch will create a master script (master.sh) will all the different batch scripts to run. For SGE_CLUSTER, each individual batch script will be created, with some standard commands to set up the infrastructure, followed by the actual commands, and at last it retrieves a Unix status code for the last command to use as a success/failure status.

You can uncomment that specific section at the end of the function: // else if ($GLB_VAR['MONITOR_TYPE']=='YOUR_CLUSTERING_TOOL') // { // /// YOUR CODE HERE // /// The goal here is to prepare the batch job // /// A master script is already opened // /// Each record in $COMMANDS contains the shell commands" for each script to run // /// You need to create a script for each record in $COMMANDS // /// The script should be saved in the jobs directory // }

You will need to modify that section to iterate over $COMMANDS, each record between the commands of one of the batch script. In each iteration you will have to:

• Create a shell script in "jobs/job_".$JOB_NUM.sh"
• Add that shell script path to the master script
• Provide any configuration you wish to provide based on your cluster application
• source setenv.sh
• execute the commands provided in the $COMMANDS record
• save the Unix script status in $W_DIR.'/jobs/status_'.$I."\n"

That's it! You can now use your cluster application to submit BioRels jobs. If you run into issues, please raise a ticket!