Adjust GPU test to work with latest nvidia daemonset on AWS/ec2

[dims]

Getting GPU test to work on AWS/EC2 as there are some flakiness on the existing CI job.

• areGPUsAvailableOnAllSchedulableNodes : ignore control plane node, it needed not have a GPU (we are not going to schedule on that node)
• makeCudaAdditionDevicePluginTestPod : usually builds a pod with 2 containers (uses 1 GPU each). As in EC2/AWS, Nodes with 2 Nvidia GPU(s) are costly, let's have a environment variable TEST_MAX_GPU_COUNT to use just 1 container (with 1 GPU) as the test workload
• reuse "NVIDIA_DRIVER_INSTALLER_DAEMONSET and point it to the latest daemonset URL needs some accomodation on the namespace logic to make it work.

See first clean run here:
https://testgrid.k8s.io/presubmits-ec2#pull-kubernetes-e2e-ec2-device-plugin-gpu&width=20

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dims

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• test/e2e/scheduling/OWNERS [dims]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dims]

/test pull-kubernetes-e2e-ec2-device-plugin-gpu

[dims]

/test pull-kubernetes-e2e-ec2-device-plugin-gpu

[dims]

/kind bug
/release-note-none
/milestone v1.30

[dims]

/kind failing-test

[dims]

/assign @tzneal @upodroid @ameukam

[dims]

/assign @pacoxu

[dims]

/test pull-kubernetes-e2e-gce-device-plugin-gpu

[dims]

/hold

we should release hold if pull-kubernetes-e2e-gce-device-plugin-gpu goes green.

[pacoxu]

/test pull-kubernetes-e2e-gce-device-plugin-gpu

after kubernetes/test-infra#32188 was merged

[dims]

/test pull-kubernetes-e2e-ec2-device-plugin-gpu

[dims]

let's make sure both the ec2 and gce jobs for device-plugin-gpu go green!

[dims]

pull-kubernetes-e2e-ec2-device-plugin-gpu is ✅

[pacoxu]

/test pull-kubernetes-e2e-gce-device-plugin-gpu

[pacoxu]

/test pull-kubernetes-e2e-ec2-device-plugin-gpu

[pacoxu]

pull-kubernetes-e2e-ec2-device-plugin-gpu is ✅

This may fixed the ec2 test.
If so, can we use https://prow.k8s.io/job-history/gs/kubernetes-jenkins/logs/ci-kubernetes-e2e-ec2-device-plugin-gpu in https://testgrid.k8s.io/sig-release-master-blocking board instead of the ci-kubernetes-e2e-gce-device-plugin-gpu

My run with #123788 failed in https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/123788/pull-kubernetes-e2e-ec2-device-plugin-gpu/1765609958725914624.

[dims]

/test pull-kubernetes-e2e-gce-device-plugin-gpu

[dims]

/skip

( skipping the pull-kubernetes-e2e-gce-device-plugin-gpu job too flaky )

[dims]

/hold cancel

[ameukam]

/lgtm

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: f3e1519b753aeb5e2679e4a2b57161fff663708c

[k8s-ci-robot]

@dims: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kubernetes-e2e-gce-device-plugin-gpu	3085765	link	false	/test pull-kubernetes-e2e-gce-device-plugin-gpu

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.