[UPSTREAM] Add check for cluster gpus before apply pod gpu config

[LaVLaS]

Add cluster gpu check before applying the gpu config to the notebook pod spec

Testing instructions:

1. Deploy RHODS with no gpu nodes present
2. Login to JH and spawn a notebook to create a JH user profile ConfigMap that stores the last selected spawner values
3. Shutdown the user notebook
4. Manually edit the JH user profile ConfigMap to add a gpu value >0 to replicate the scenario where the previous user notebook pod had gpus attached
5. Login to the JH Spawner UI and confirm that there is no UI element for selecting gpu and attempt to spawn a notebook
6. Confirm that the user notebook pod is scheduled anddoes not request any gpus. .spec.resources.limits['nvidia.com/gpu'] should not exist

You can verify that this does not break existing functionality with the following steps.

1. Deploy RHODS + GPU Operator
2. Add a GPU node to the cluster and wait for the gpu operator to enable the GPUs (gpu node has label nvidia.com/gpu: X)
3. Login to JH and spawn a notebook that requests a gpu
4. Wait for the notebook to spawn successfully and shut it down
5. Remove the GPU node from the cluster wait until there are no GPUs in the cluster
6. Go to the JH spawner and verify that there is no UI element for requesting GPUs and start any notebook
7. Confirm that the notebook pod spawns successfully with no GPUs requested

Signed-off-by: Landon LaSmith LLaSmith@redhat.com

• The commits are squashed in a cohesive manner and have meaningful messages.
• For commits that came from upstream, [UPSTREAM] has been prepended to the commit message
• JIRA link(s): RHODS-3069
• The Jira story is acked
• An entry has been added to the latest build document in Build Announcements Folder.
• Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious)

[VaishnaviHire]

@LaVLaS Can you share the link to the live-build ?

[VaishnaviHire]

This did not work for me on quay.io/modh/rhods-operator-live-catalog:1.9.0-rhods-3059 live-build. I was unable to schedule a notebook pod after updating the user profile configmap.

data:
  profile: |
    gpu: '1'
    last_selected_image: s2i-minimal-notebook:py3.8-rhodsversion
    last_selected_size: Default

Notebook pod:

  containers:
    - resources:
        limits:
          cpu: '2'
          memory: 8Gi
          nvidia.com/gpu: '1'
        requests:
          cpu: '1'
          memory: 4Gi
          nvidia.com/gpu: '1'

[LaVLaS]

Live build avaiable at quay.io/llasmith/rhods-operator-live-catalog:1.9.0-rhods-3069

[VaishnaviHire]

/lgtm

Worked well with the updated build.