jupyter-web-app throws KeyError: 'matchLabels' when PodDefault does not contain matchLabels selector

[nichoio]

/kind bug

What steps did you take and what happened:
This is half bug report, half question.
I'm looking for a way to automatically apply tolerations to notebooks, pipelines and pods in general which belong to the namespace owned by some user - without the need of the user to select anything in spawner UI form. I'm trying to abstract away taints and tolerations as much as possible.

Since I'm on AWS EKS and PodTolerationRestriction are not available there, I have looked into PodDefaults to achieve this.
PodDefaults seem to accept matchLabels as well as matchExpressions selectors. I have also noticed that KF seems to apply label notebook-name to every new notebook pod - so far so great. I can apply matchExpressions like this:

spec:
  desc: Some poddefault
  selector:
    matchExpressions:
      - { key: notebook-name, operator: Exists}  # matches any notebook
#      - { key: foo, operator: NotIn, values: [ bar ] }  # matches anything in this namespace

This works in general. Any notebook started will receive this default. Or alternatively, when using { key: foo, operator: NotIn, values: [ bar ] } , any pod in this namespace receives the default. It's certainly a hack and I wonder to what degree this is intended behavior.
Also note that users don't have to set anything in the spawner GUI to enable this which is what I intend.

What did you expect to happen:
Now the part that doesn't work smoothly. When omitting matchLabels in PodDefault selector, jupyter-web-app throws an error when entering spawner GUI:

2022-12-06 15:47:23,550 | kubeflow.kubeflow.crud_backend.errors.handlers | ERROR | 'matchLabels'
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1820, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1796, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/src/apps/common/routes/get.py", line 32, in get_poddefaults
    label = list(pd["spec"]["selector"]["matchLabels"].keys())[0]
KeyError: 'matchLabels'

Looks like the code expects some matchLabels selector. As a result I assume, Configurations drop down menu of spawner doesn't display anything but spawner form can still be submitted.

My question would be if there is any way to apply a PodDefault without this PodDefault showing up in spawner UI? Maybe there is something that could be done with the content of spawner_ui_config.yaml ? Thanks.

Environment:

• Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard): 1.6

• kfctl version: (use kfctl version): not applicable

• Kubernetes platform: AWS EKS

• Kubernetes version: 1.23

• OS (e.g. from /etc/os-release): Amazon Linux

• Jupyter web app image: docker.io/kubeflownotebookswg/jupyter-web-app:v1.6.1

[elanv]

Is there anything updated?

[danvaida]

I'd also be interested in finding out a way to make use of the taints I got in my clusters. IMO, being able to specify things like nodeSelectors and tolerations should be imperative, for a stack that's effectively advertised as "the K8s way for ML".

It's clear from the docs that Kubeflow expects to be the one and only thing running on our K8s clusters but that's not exactly feasible for most of us.

As we can see, leveraging K8s' mechanisms in order to achieve this segregation for all intents and purposes, renders some of the components that rely on those selectors as broken. Therefore, ideally, we would be able to set those tolerations on all the existing and new pods.

[adgaultier]

Any update?
More context:
/jupyter/api/namespaces/xxx/poddefaults seem to be only working with podDefaults using MatchLabels as it is hard-coded here

but PodDefault uses metav1.LabelSelectorAsSelector which is compatible with (more expressive) MatchExpressions (cf here)

[boarder7395]

@danvaida I'm not sure I follow the issue you're having. Tolerations and nodeselectors should work with poddefaults. I've added a PR that should fix the issue with matchExpressions since I ran into this issue today as well. I built the image from my fork and deployed on my clusters and this has fixed the issue for me.

[ca-scribner]

I think the root cause of this is #7552, where the 0th matchLabel is being used to "label" each PodDefault instead of the PodDefault's name