Allow the usage of a custom docker image for K8s driver pod

[micans]

The short: It would be great if we can specify a docker image to use for nextflow kuberun.

The long: We have created a nextflow image with additonal software so that we can log in to the pod and have some tools available. This has worked extremely well and was simple to set-up; we have a question about the configuration file set-up; especially about the persistent volume claim setting and the role of /etc/nextflow/nextflow.config. We create the pod using a yaml file obtained from a nextflow kuberun invocation. This is it:

apiVersion: v1
kind: Pod
metadata:
  name: king-fisher
  namespace: default
  labels: {app: nextflow, runName: king-fisher}
spec:
  restartPolicy: Never
  containers:
  - name: king-fisher
    image: quay.io/cellgeni/nf-login:v0.32.0u1-a
    command: [/bin/bash, -c, source /etc/nextflow/init.sh; tail -f /dev/null]
    workingDir: /mnt/gluster/svd
    env:
    - {name: NXF_WORK, value: /mnt/gluster/svd/work}
    - {name: NXF_ASSETS, value: /mnt/gluster/projects}
    - {name: NXF_EXECUTOR, value: k8s}
    volumeMounts:
    - {name: vol-1, mountPath: /mnt/gluster}
    - {name: vol-2, mountPath: /etc/nextflow}
  volumes:
  - name: vol-1
    persistentVolumeClaim: {claimName: nf-pvc}
  - name: vol-2
    configMap: {name: nf-config-8f8cdc01}

This config map was created by an earlier run of nextflow kuberun. It looks like this:

bc-2,cellgeni/nf-login (master *%), kubectl describe configmap nf-config-8f8cdc01
Name:         nf-config-8f8cdc01
Namespace:    default
Labels:       <none>
Annotations:  <none>

Data
====
init.sh:
----
mkdir -p '/mnt/gluster/svd'; if [ -d '/mnt/gluster/svd' ]; then cd '/mnt/gluster/svd'; else echo 'Cannot create nextflow userDir: /mnt/gluster/svd'; exit 1; fi; [ -f /etc/nextflow/scm ] && ln -s /etc/nextflow/scm $NXF_HOME/scm; [ -f /etc/nextflow/nextflow.config ] && cp /etc/nextflow/nextflow.config $PWD/nextflow.config; 
nextflow.config:
----
k8s {
   debug {
      yaml = true
   }
   storageClaimName = 'nf-pvc'
   storageMountPath = '/mnt/gluster'
}

process {
   executor = 'k8s'
}

cacheable = true
runName = 'nasty-cajal'

Events:  <none>

We create a nextflow pod with kubectl create -f our-copied-file.yaml.
When we log in to the pod with kubectl exec -it king-fisher bash, the file in /etc/nextflow/nextflow.config is this:

k8s {
   debug {
      yaml = true
   }
   storageClaimName = 'nf-pvc'
   storageMountPath = '/mnt/gluster'
}

process {
   executor = 'k8s'
}

cacheable = true
runName = 'nasty-cajal'

The environment settings are this:

(king-fisher|svd/bhsi) env | grep NXF
NXF_OPTS=-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap
NXF_EXECUTOR=k8s
NXF_ASSETS=/mnt/gluster/projects
NXF_WORK=/mnt/gluster/svd/work
NXF_HOME=/.nextflow

Now, nextflow runs fail without producing an exit code and a message that the external system does not work. We assume this has to do with the persistent volume claim; if we add this nextflow.config file, repeating information from /etc/nextflow/nextflow.config, then nextflow works:

k8s {
   storageClaimName = 'nf-pvc'
   storageMountPath = '/mnt/gluster'
}

We were looking for a way to make this cleaner, but now the first thing that comes to mind is that nextflow already takes care of everything; we think perhaps as a generalisation there could be an option where a docker image can be specified.

[pditommaso]

This makes sense and it's quite easy to implement. Currently the image named is embedded in the K8sDriverLauncher.

The idea could be to add a cli option to CmdKubeRun command.

[pditommaso]

Included in version 18.12.0-edge.

[micans]

Thanks a lot @pditommaso !