Add PollImmediate for e2e metrics to avoid race with prom scrape interval

[enxebre]

What this PR does / why we need it:
We have seen failures Failed to validate that metrics are exposed: "hypershift_cluster_deletion_duration_seconds" not found for the TestNodePool/ValidateMetricsAreExposed

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_hypershift/2477/pull-ci-openshift-hypershift-main-e2e-aws/1651173441761447936/artifacts/e2e-aws/run-e2e/artifacts/TestNodePool_PreTeardownClusterDump/namespaces/hypershift/core/pods/logs/operator-85bbc866c7-7kjxl-operator.log

Even though the code for exposing the metric has run as proved by this log

{"level":"info","ts":"2023-04-26T11:41:27Z","msg":"Deleted hostedcluster","controller":"hostedcluster","controllerGroup":"hypershift.openshift.io","controllerKind":"HostedCluster","hostedCluster":{"name":"example-gxnf4","namespace":"e2e-clusters-6t9j8"},"namespace":"e2e-clusters-6t9j8","name":"example-gxnf4","reconcileID":"346f2e3c-6cab-41d8-934d-1783e5ae267e","name":{"namespace":"e2e-clusters-6t9j8","name":"example-gxnf4"}}

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_hypershift/2477/pull-ci-openshift-hypershift-main-e2e-aws/1651173441761447936/artifacts/e2e-aws/run-e2e/artifacts/TestNodePool_DestroyCluster_1/namespaces/hypershift/core/pods/logs/operator-85bbc866c7-7kjxl-operator.log

Which issue(s) this PR fixes (optional, use fixes #<issue_number>(, fixes #<issue_number>, ...) format, where issue_number might be a GitHub issue, or a Jira story:
Fixes #

Checklist

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: enxebre

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [enxebre]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[muraee]

/lgtm

[jparrill]

/lgtm

[davidvossel]

/hold

Can you remove the skip for none platform to ensure this addresses that use case? Or is that not part of the goal here?

		if hc.Spec.Platform.Type == hyperv1.NonePlatform {
			t.Skip()
		}

feel free to remove the hold if this isn't the goal, i just wanted to make sure we got the right signal we were looking for with the e2e tests

[enxebre]

feel free to remove the hold if this isn't the goal, i just wanted to make sure we got the right signal we were looking for with the e2e tests

Thanks @davidvossel I don't want to take any risk of conflating potentially different root causes. Let me ship this as it's if it passes and I'll immediately follow up to remove the none skip.
/hold cancel

[sjenning]

/lgtm

[sjenning]

FYI: issue this should fix took out an e2e

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-hypershift-release-4.14-periodics-e2e-aws-ovn/1651252706251116544

EnsureAPIBudget was also part of the flake.

[imain]

/lgtm

[openshift-ci]

@enxebre: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.