k8s nodeSelector

[KevinSayers]

New feature

Currently Nextflow does not provide a way to specify nodes within a cluster to run jobs on. Like with other executors it would be beneficial to be able to tailor task execution to specific nodes with potentially varied hardware.

This is related to some aspects of #785

Usage scenario

Running Nextflow on a heterogeneous cluster and having a process that requires a high memory. Specify in the process directives either in the main Nextflow script or in the config which nodes can handle this within a k8s cluster.

Suggest implementation

Add a nodeSelector option to the pod directive scope.

nodeSelector docs: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector

process foo{
   pod [node: 'cloud.google.com/gke-nodepool', value: 'highmemory-pool']
   ....
}

or alternatively make an independent directive for nodeSelector

process foo{
   nodeSelector [label: 'cloud.google.com/gke-nodepool', value: 'highmemory-pool']
.....
}

An example pod definition with the nodeSelector

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    env: test
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
  nodeSelector:
    cloud.google.com/gke-nodepool: highmemory-pool

[pditommaso]

This is very interesting and matches with a similar request other schedulers #911. I think we could introduce a nodes or affinity directive to allow user to specify a node selection rule depending the underlying platform.

[pditommaso]

The above commit adds the ability to specify the node selector in the pod directive. The selector is specified as a key-value string pair separated by a equals character, multiple key-value pairs can be specified separating them with a comma, eg:

process foo{
   pod nodeSelector: 'cloud.google.com/gke-nodepool=highmemory-pool'

...
}

(fixed the wrong syntax pod:)

[pditommaso]

Included in version 18.12.0-edge.

[cbmckni]

Just an update for others seeing this issue, the syntax

pod: nodeSelector: 'cloud.google.com/gke-nodepool=highmemory-pool'

is incorrect. The correct syntax is:

pod nodeSelector: 'cloud.google.com/gke-nodepool=highmemory-pool'

Refer to https://www.nextflow.io/docs/latest/process.html#pod