Add autoscale check for GPU

[maroroman]

Resolves #407

Description

This PR adds logic to check the MachineAutoscaler CRs for the maximum amount of scalable GPU nodes.

How Has This Been Tested?

Testing was done on a OSD cluster with a gpu node with autoscaling enabled.

1. Use following dashboard image: quay.io/mroman_redhat/odh-dashboard:dev-gpu-3
2. The dashboard detects all MachineAutoscaler CRs and loops through their MachineSets which contain the machine.openshift.io/GPU value which should contain the true gpu amount (correct me if my assumption is wrong)

Resulting backend response was this:

The value is used by the frontend to extend the dropdown if the scalable gpus are over the amount of gpus reported by prometheus metrics.

Merge criteria:

• The commits are squashed in a cohesive manner and have meaningful messages.
• Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
• The developer has manually tested the changes and verified that the changes work

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[maroroman]

@andrewballantyne Should I add the autoscaler value to the available gpus and let the dropdown just show a higher number without a UX difference?

[andrewballantyne]

@maroroman see Jeff's comment here, I think the answer to your question is this sentence:

The number in the drop down should be the max available on any single node

No UX difference, just show the numbers.

[maroroman]

FYI: Added one more force push to deal with a log and fix the role yaml.

[andrewballantyne]

Added some data caching on the backend and removed the re-fetching toggle on the frontend.

[cfchase]

@maroroman we've made some progress here, but please test with a built image:

1. Can schedule a multi gpu notebook when a multi-gpu node is scaled down to 0 and there are no other gpu nodes but triggers a scaling event.
2. Can start a multi-gpu notebook when the available node only has 1 gpu available
3. Can start a single gpu notebook when 1 is available without scaling up

[cfchase]

Some comments on existing code:

There's another await in a for loop you need to Promise.all:
https://github.com/opendatahub-io/odh-dashboard/pull/573/files#diff-fe048868b84dce849bd698d4f237f445ecc70661f9ca1842d9a9429b4c2c37feR50

If that requests takes 3 seconds and there's 10 nodes, the user is going to be waiting for 30 seconds for the dialog to get a gpu value. It will grow O(n) as the cluster increases nodes. I think this is affecting all current users. Unless a request needs a previous result you shouldn't wait before executing.

I think you also have to take out the await here: https://github.com/opendatahub-io/odh-dashboard/pull/573/files#diff-fe048868b84dce849bd698d4f237f445ecc70661f9ca1842d9a9429b4c2c37feR78

This token should be found at start up, added to the fastify.kube.saToken so you don't have to fetch it on every request
https://github.com/opendatahub-io/odh-dashboard/pull/573/files#diff-fe048868b84dce849bd698d4f237f445ecc70661f9ca1842d9a9429b4c2c37feR40

[cfchase]

/lgtm

[andrewballantyne]

/unhold

Forgot to unhold this after Chris approved it. Doing so now.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andrewballantyne, cfchase

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [andrewballantyne]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment