Swap gce-conformance on sig-release-master-blocking from kubetest to kubetest2

[amwat]

First of many! No bootstrap, no scenarios, no kubetest :)

xref: kubernetes/enhancements#2464

The kubetest2 gce-conformance job has been stable for a while:
https://testgrid.k8s.io/conformance-all#Conformance%20-%20GCE%20-%20master%20-%20kubetest2

the old job will still be visible at: https://testgrid.k8s.io/conformance-all#Conformance%20-%20GCE%20-%20master
will not be turned down yet

Only targetting master, older release branches will still use kubetest based jobs.

/cc @BenTheElder @spiffxp
/cc @kubernetes/ci-signal
/cc @kubernetes/sig-release

[spiffxp]

/area conformance
/cc @dims @johnbelamaric
as FYI

I compared results of latest runs

• latest-kubetest2
• latest

The difference in number of tests passed is due to the e2e.go suite having 13 steps, where kubetest2's (unamed) suite has 4 steps

Comparing durations on testgrid:

• latest-kubetest2
• latest

Overall is about 10min longer for kubetest2, but Test is about 10min shortee for kubetest2. Turns out this is due to the fact that there is a kubetest2.Build stage.

The kubetest2 job hasn't been passing for a full two weeks, but I see nothing in its history to indicate that it's providing lower quality signal than the job it's replacing.

So, suggestions for followup:

• kubetest2 job should download pre-built binaries from k8s-release-dev bucket
• kubetest2 should give its "meta" suite a name, maybe kubetest2

/approve
/lgtm
/hold
for comment, I'm gonna go ping release team

[spiffxp]

/assign @thejoycekung @palnabarun
Release team: CI Signal lead, Release lead
/assign @hasheddan @jeremyrickard
SIG Release TLs

I'll remove /hold If no objections by SIG Testing meeting tomorrow at 10am PT, or +1's from at least one of each /assign line

[hasheddan]

/lgtm

Thanks @amwat!

[amwat]

+1 ack on the differences

• Adding support for GCE extract ([GCE] add support for extracting pre-existing builds from GCS kubernetes-sigs/kubetest2#106) (although master job should ideally be building from master? 🤔 )
• testsuite name was added in Add testsuite name in the generated junit-runner.xml file kubernetes-sigs/kubetest2#93 and it shows up as such in testgrid, need to figure out why it's not showing up in the junit lens

[BenTheElder]

Adding support for GCE extract (kubernetes-sigs/kubetest2#106) (although master job should ideally be building from master? 🤔 )

No, the vast majority of jobs targeting any branch use the existing uploads of this. It adds a little latency (the time it takes us to make quick-release + upload) but saves us a lot of compute, we have a continuous build to consume.

(I need to fix this for KIND, but it was not subsuming an existing job, and for some time we were intentionally using different builds in some jobs).

[chizhg]

• testsuite name was added in kubernetes-sigs/kubetest2#93 and it shows up as such in testgrid, need to figure out why it's not showing up in the junit lens

Do you have a spyglass link for this? I checked one of our test runs and the kubetest2 testsuites looks good in the junit lens.

[thejoycekung]

/lgtm

The job looks stable, and I think it's super awesome we're moving away from bootstrap. Thanks y'all!

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: amwat, hasheddan, spiffxp, thejoycekung

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• config/jobs/kubernetes/sig-cloud-provider/OWNERS [spiffxp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[amwat]

Do you have a spyglass link for this? I checked one of our test runs and the kubetest2 testsuites looks good in the junit lens.

tracked in kubernetes-sigs/kubetest2#107

[spiffxp]

/hold cancel
Thanks for the review all!

[k8s-ci-robot]

@amwat: Updated the job-config configmap in namespace default at cluster test-infra-trusted using the following files:

• key gce-conformance.yaml using file config/jobs/kubernetes/sig-cloud-provider/gcp/gce-conformance.yaml

In response to this:

First of many! No bootstrap, no scenarios, no kubetest :)

xref: kubernetes/enhancements#2464

The kubetest2 gce-conformance job has been stable for a while:
https://testgrid.k8s.io/conformance-all#Conformance%20-%20GCE%20-%20master%20-%20kubetest2

the old job will still be visible at: https://testgrid.k8s.io/conformance-all#Conformance%20-%20GCE%20-%20master
will not be turned down yet

Only targetting master, older release branches will still use kubetest based jobs.

/cc @BenTheElder @spiffxp
/cc @kubernetes/ci-signal
/cc @kubernetes/sig-release

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.