[openmpi]- NodeSelector not working.

[karthikvadla]

As mentioned here in GPU Training section trying to use --nodeSelector to specify pool of nodes to launch my workloads.

But --nodeSelector is not working as expected.

--nodeSelector=<nodeSelector> Comma-delimited list of "key=value" pairs to select the worker nodes. e.g. "cloud.google.com/gke-accelerator=nvidia-tesla-k80" [default: null, type: string]

Above description is bit confusing, mentioned type as string but when explaining written as comma -delimited list (Typo need to be corrected)

Tried as below.

if type is array
NODE_SELECTOR='["kubernetes.io/hostname=gke-x1", "kubernetes.io/hostname=gke-x2"]' - It didn't honor, randomly picked up some nodes.

or
if type is string
NODE_SELECTOR="kubernetes.io/hostname=gke-x1, gke-x2". It gave me below error

level=error msg="find objects: RUNTIME ERROR: Index 1 out of bounds, not within [0, 1)\n\t/Users/kvadla/git-repo/mlt/examples/horovod_unet/mlt_app/unet-horovod-2e6fbba6-7e39-41f9-98b5-1f3241001e33/vendor/kubeflow/openmpi/util.libsonnet:14:46-81\tobject \n\t:1222:30-34\tthunk from <thunk from <function >>\n\t:1216:24-25\tthunk from <thunk from <function >>\n\t\tbuiltin function \n\t:1217:8-9\tfunction \n\t:1165:25-26\tthunk from <thunk from <function >>\n\t\tbuiltin function \n\t:1167:29-31\tthunk from <function >\n\t\tbuiltin function \n\t:1167:8-36\tfunction \n\t\t\n\t:1222:20-35\tthunk from <function >\n\t:1208:10-11\tfunction \n\t:1165:25-26\tthunk from <thunk from <function >>\n\t\tbuiltin function \n\t:1167:29-31\tthunk from <function >\n\t\tbuiltin function \n\t:1167:8-36\tfunction \n\t\t\n\t:1222:10-36\tfunction \n\t...\n\t\tbuiltin function \n\t:1167:29-31\tthunk from <function >\n\t\tbuiltin function \n\t:1167:8-36\tfunction \n\t\t\n\t:1218:35-56\tfunction \n\t\tbuiltin function \n\t:1222:20-35\tthunk from <function >\n\t:1208:10-11\tfunction \n\t:1165:25-26\tthunk from <thunk from <function >>\n\t\tbuiltin function \n\t:1167:29-31\tthunk from <function >\n\t\tbuiltin function \n\t:1167:8-36\tfunction \n\t\t\n\t:1222:10-36\tfunction \n\t\tbuiltin function \n\t\tbuiltin function <$objectFlatMerge>\n\textvar:__ksonnet/components:7:3-58\tobject \n\tDuring manifestation\t\n"
make: *** [deploy] Error 1

Found out during discussion #838
@everpeace

[karthikvadla]

/area openmpi
/priority p1

[everpeace]

/cc @jiezhang
Would you mind taking a look at this??

[jiezhang]

Sorry about the confusion. The syntax is comma separated list of key=value pairs, e.g. "kubernetes.io/hostname=gke-x1,cloud.google.com/gke-accelerator=nvidia-tesla-k80". The conditions are AND'ed together.

Unfortunately we don't support IN operator in the expression at the moment. We need to figure out how to support IN operator here: https://github.com/kubeflow/kubeflow/blob/master/kubeflow/openmpi/util.libsonnet

One workaround I can think of is to label your nodes such that gke-x1 and gke-x1 share one common lable and use that as NODE_SELECTOR (e.g. "node-type=gke").

[jiezhang]

If you're using gcloud, you can specify node labels this way:

gcloud container node-pools create ... --node-labels=node-type=gke

[karthikvadla]

Oh yeah my nodes are already created. Will try to update lables for those nodes and use it.
Thanks @jiezhang

[jiezhang]

/assign @jiezhang

[ashahba]

The nodes are labeled now and we have:

$ kubectl get nodes --selector=node-type=highmem
NAME                                            STATUS    ROLES     AGE       VERSION
gke-xxx-xx-xxxxxxxx-xxxx   Ready     <none>    53d       v1.10.2-gke.3
gke-yyy-yy-yyyyyyyy-yyyy   Ready     <none>    53d       v1.10.2-gke.3
gke-zzz-zz-zzzzzzzz-zzzz   Ready     <none>    53d       v1.10.2-gke.3

[jiezhang]

Looks like NodeSelector does not support IN operator. We need to support Affinity if you want "hostname IN (gke-x1, gke-x2)".

I'm closing this for now. Please open a feature request if you need the feature.

/close