Add renku cluster commands

[TaoSunVoyage]

To solve the issue raised in SwissDataScienceCenter/renku#1929, we propose to have renku cluster commands to submit renku workflows as batch jobs on the HPC systems:

• renku cluster config: set options/variables for SLURM scripts in .renku/renku.ini

• renku cluster run/update/rerun: submit renku run/update/rerun as a batch job to the HPC systems

Typical use case:

# User is in the HPC environment
# User has installed renku on the HPC
# for which user needs to install/update git-lfs and nodejs

# User clones the project
renku clone <renkulab-project.git>
cd <renkulab-project>

# User sets options/variables used in SLURM scripts, for example
renku cluster config sbatch.partition debug # Options for sbatch
renku cluster config renku.image <image_uri_in_renkulab-registry> # Variables used in the sbatch scripts

# User submits batch jobs
renku cluster update --all # which will create a batch job with chosen image based on some templates (see below)

Example template of SBATCH script with Singularity:

#!/bin/bash -l
#SBATCH --{{ sbatch.option1.key }}={{ sbatch.option1.value }}
#SBATCH --{{ sbatch.option2.key }}={{ sbatch.option2.value }}

cd ${SCRATCH}

export RENKU_IMAGE=docker://{{ renku.image }}

# Token for Git-push (personal access token or project access token)
export RENKU_TOKEN={{ renku.token }}

export RENKU_PROJ_GIT=https://oauth2:${RENKU_TOKEN}@{{ renku.git }}

export RENKU_PROJ_NAME={{ renku.proj_name }}

export GIT_AUTHOR_NAME={{ renku.git_username }}
export EMAIL={{ renku.git_email }}

cat << END > ${HOME}/runs/${SLURM_JOB_ID}-run_HPC.sh
#!/bin/bash
set -e

# overwrite HOME & PATH
HOME=/home/\${USER}
PATH=\${PATH}:\${HOME}/.local/bin

# check env
which python

# clone the project
cd \${SCRATCH}
rm -rf \${RENKU_PROJ_NAME}
renku clone \${RENKU_PROJ_GIT}
cd \${RENKU_PROJ_NAME}

# renku commands
{{ renku.commands }} # <-- renku commands to run in HPC environment
renku save

# clean up
cd ..
rm -rf \${RENKU_PROJ_NAME}

rm ${HOME}/runs/${SLURM_JOB_ID}-run_HPC.sh
END

srun singularity run -C -B ${SCRATCH} ${RENKU_IMAGE} bash ${HOME}/runs/${SLURM_JOB_ID}-run_HPC.sh

Image used in batch execution

Default images in renkulab are in general very large due to jupyter and conda stack, etc. We suggest to use a batch version of image as descirbed in SwissDataScienceCenter/renku-project-template#118