Add periodic conformance test run against cloud-provider-gcp using kubetest2 #18714

michaelmdresser · 2020-08-06T22:43:09Z

Adds a new periodic job to run the conformance ginkgo suite against the cloud-provider-gcp repository. Test job is based on
ci-kubernetes-gce-conformance-latest, with the main difference being using the new kubetest2 GCE deployer instead of kubetest. Because cloud-provider-gcp does not have published release tars, this job has to clone and build the cloud-provider-gcp repository. The job also does not include the forking and the alerting in the job it is based on because it needs to be validated in production prow before meaningful decisions can be made using it.

Additional note: due to #18706, this job uses KRTE instead of kubekins-e2e as the test runner.

Tested in a GKE cluster using mkpj and mkpod.

Adds a new periodic job to run the conformance ginkgo suite against the cloud-provider-gcp repository. Test job is based on ci-kubernetes-gce-conformance-latest, with the main difference being using the new kubetest2 GCE deployer instead of kubetest. Because cloud-provider-gcp does not have published release tars, this job has to clone and build the cloud-provider-gcp repository. The job also does not include the forking and the alerting in the job it is based on because it needs to be validated in production prow before meaningful decisions can be made using it.

michaelmdresser · 2020-08-06T22:54:29Z

config/jobs/kubernetes/cloud-provider-gcp/cloud-provider-gcp-periodics.yaml

+    preset-service-account: "true"
+  spec:
+    containers:
+    - image: gcr.io/k8s-testimages/krte:v20200806-28035c4-1.16


I'm not sure which tag of this KRTE release I should be using. This is the postsubmit that built it.

we need your fix so this is fine, but eventually we should use the auto generated tag.

michaelmdresser · 2020-08-06T22:54:57Z

config/jobs/kubernetes/cloud-provider-gcp/cloud-provider-gcp-periodics.yaml

@@ -0,0 +1,32 @@
+periodics:
+- interval: 3h


Maybe this interval should be longer. Conformance tests take a while.

3h seems fine, that's also the cadence for the existing job.

michaelmdresser · 2020-08-06T22:56:49Z

config/jobs/kubernetes/cloud-provider-gcp/cloud-provider-gcp-periodics.yaml

+        go get sigs.k8s.io/kubetest2@latest;
+        go get sigs.k8s.io/kubetest2/kubetest2-gce@latest;
+        go get sigs.k8s.io/kubetest2/kubetest2-tester-ginkgo@latest;
+        kubetest2 gce -v 2 --repo-root $REPO_ROOT --build --up --down --test=ginkgo -- --focus-regex='\[Conformance\]' --parallel=30


The original job doesn't configure a parallel amount for the tester. Doing so speeds up tests. I would appreciate perspective on potential issues this could cause.

Not all jobs can be safely run in parallel and usually causes failures as compared to when run serially.
Let's stick to serial for now, or we can have another job for parallel.

Dropped the parallel

amwat

Really excited to see this.

/cc @cheftako @cici37

amwat · 2020-08-06T23:19:40Z

config/jobs/kubernetes/cloud-provider-gcp/cloud-provider-gcp-periodics.yaml

+    preset-service-account: "true"
+  spec:
+    containers:
+    - image: gcr.io/k8s-testimages/krte:v20200806-28035c4-1.16


we need your fix so this is fine, but eventually we should use the auto generated tag.

amwat · 2020-08-06T23:19:42Z

config/jobs/kubernetes/cloud-provider-gcp/cloud-provider-gcp-periodics.yaml

@@ -0,0 +1,32 @@
+periodics:
+- interval: 3h


3h seems fine, that's also the cadence for the existing job.

amwat · 2020-08-06T23:20:46Z

config/jobs/kubernetes/cloud-provider-gcp/cloud-provider-gcp-periodics.yaml

+        go get sigs.k8s.io/kubetest2@latest;
+        go get sigs.k8s.io/kubetest2/kubetest2-gce@latest;
+        go get sigs.k8s.io/kubetest2/kubetest2-tester-ginkgo@latest;
+        kubetest2 gce -v 2 --repo-root $REPO_ROOT --build --up --down --test=ginkgo -- --focus-regex='\[Conformance\]' --parallel=30


Not all jobs can be safely run in parallel and usually causes failures as compared to when run serially.
Let's stick to serial for now, or we can have another job for parallel.

amwat · 2020-08-06T23:22:41Z

config/jobs/kubernetes/cloud-provider-gcp/cloud-provider-gcp-periodics.yaml

+  cluster: k8s-infra-prow-build
+  name: ci-cloud-provider-gcp-conformance-latest
+  annotations:
+    testgrid-tab-name: Conformance - Cloud Provider GCP - master


you'll need a dashboard here: https://github.com/kubernetes/test-infra/blob/master/config/testgrids/kubernetes/sig-cloud-provider/gcp/config.yaml

Added, hopefully that's the correct syntax.

amwat · 2020-08-06T23:45:49Z

config/jobs/kubernetes/cloud-provider-gcp/cloud-provider-gcp-periodics.yaml

+        set -o nounset;
+        set -o pipefail;
+        set -o xtrace;
+        git clone https://github.com/kubernetes/cloud-provider-gcp;


can we use extra_refs instead of manually git cloning.

Periodics don't get access to extra_refs. I believe this is also indicated by the fact that there is no clonerefs containers as part of periodic job specs.

I'm not sure if that's true. extra_refs is a ProwJob field so it's independent of whether it's a presubmit or a periodic.
We use it in a bunch of jobs

test-infra/config/jobs/kubernetes-sigs/kind/kind.yaml

Lines 5 to 9 in 70c4596

extra_refs:

- org: kubernetes-sigs

repo: kind

base_ref: master

path_alias: sigs.k8s.io/kind

Hmm. I tried adding it to my periodic, regenerating it, and then looking for the repo and couldn't find it.

you'll need decorate: true

decorate: true solves this problem. Updated the PR to use extra_refs to get the cloud-provider-gcp repo.

amwat · 2020-08-06T23:48:17Z

config/jobs/kubernetes/cloud-provider-gcp/cloud-provider-gcp-periodics.yaml

+        cd cloud-provider-gcp;
+        REPO_ROOT=$(git rev-parse --show-toplevel);
+        cd;
+        go get -u github.com/onsi/ginkgo/ginkgo;


fyi the ginkgo binary also exists in the same tarball as e2e.test .
Let's start with this.
and as a follow up add the logic inside the tester to use that one and then remove this from here.

amwat

/lgtm
/approve

amwat · 2020-08-07T19:07:21Z

/label tide/merge-method-squash

amwat · 2020-08-07T19:08:23Z

/cc @BenTheElder @cheftako
approval for adding the dashboard

spiffxp · 2020-08-07T19:29:54Z

config/jobs/kubernetes/cloud-provider-gcp/cloud-provider-gcp-periodics.yaml

+          cpu: 2000m
+          memory: 2G
+        requests:
+          cpu: 2000m
+          memory: 2G


The closest analogue I would have for this is maybe pull-kubernetes-e2e-gce, which is currently at cpu: 4, memory: 14Gi.

Can be tuned once job is running regularly and can be inspected.

amwat · 2020-08-07T19:34:44Z

/lgtm
@spiffxp can you approve the dashboard changes as well?

spiffxp

/approve
for the testgrid changes as well
/lgtm

k8s-ci-robot · 2020-08-07T19:35:08Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: amwat, michaelmdresser, spiffxp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~config/jobs/kubernetes/cloud-provider-gcp/OWNERS~~ [amwat,spiffxp]
~~config/testgrids/kubernetes/sig-cloud-provider/OWNERS~~ [spiffxp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2020-08-07T20:10:03Z

@michaelmdresser: Updated the job-config configmap in namespace default at cluster default using the following files:

key cloud-provider-gcp-periodics.yaml using file config/jobs/kubernetes/cloud-provider-gcp/cloud-provider-gcp-periodics.yaml

In response to this:

Adds a new periodic job to run the conformance ginkgo suite against the cloud-provider-gcp repository. Test job is based on
ci-kubernetes-gce-conformance-latest, with the main difference being using the new kubetest2 GCE deployer instead of kubetest. Because cloud-provider-gcp does not have published release tars, this job has to clone and build the cloud-provider-gcp repository. The job also does not include the forking and the alerting in the job it is based on because it needs to be validated in production prow before meaningful decisions can be made using it.

Additional note: due to #18706, this job uses KRTE instead of kubekins-e2e as the test runner.

Tested in a GKE cluster using mkpj and mkpod.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

liggitt · 2020-08-17T14:10:46Z

it looks like this hasn't passed since it was added: https://testgrid.k8s.io/provider-gcp-periodics#Conformance%20-%20Cloud%20Provider%20GCP%20-%20master&width=5

the failing tests look exclusively connected to the new APIs added in 1.19 (ingress v1, ingressclass v1, certificatesigningrequests v1)... what configuration is this running with, and are we sure it's running against the 1.19 artifacts?

liggitt · 2020-08-17T14:29:56Z

opened kubernetes/kubernetes#94051 to track

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Aug 6, 2020

k8s-ci-robot requested review from amwat and mikedanese August 6, 2020 22:43

michaelmdresser commented Aug 6, 2020

View reviewed changes

amwat reviewed Aug 6, 2020

View reviewed changes

k8s-ci-robot requested review from cheftako and cici37 August 6, 2020 23:23

Dropped parallel to 1

5db8fdc

michaelmdresser force-pushed the c-p-gcp-conformance-kt2 branch from 6d579ba to 5db8fdc Compare August 6, 2020 23:31

Added dashboard for cloud-provider-gcp periodics

f650163

k8s-ci-robot added area/provider/gcp Issues or PRs related to gcp provider area/testgrid labels Aug 6, 2020

amwat reviewed Aug 6, 2020

View reviewed changes

michaelmdresser added 2 commits August 7, 2020 16:55

Added container limits and requests for conformance periodic

d0338e5

Use extra_refs to get cloud-provider-gcp repo

8bec7c1

amwat reviewed Aug 7, 2020

View reviewed changes

k8s-ci-robot assigned amwat Aug 7, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 7, 2020

k8s-ci-robot added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Aug 7, 2020

k8s-ci-robot requested a review from BenTheElder August 7, 2020 19:08

spiffxp reviewed Aug 7, 2020

View reviewed changes

Bumped resources for conformance test runner

b6a5dc0

Can be tuned once job is running regularly and can be inspected.

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 7, 2020

spiffxp approved these changes Aug 7, 2020

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 7, 2020

k8s-ci-robot assigned spiffxp Aug 7, 2020

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 7, 2020

k8s-ci-robot merged commit 78767f2 into kubernetes:master Aug 7, 2020

k8s-ci-robot added this to the v1.19 milestone Aug 7, 2020

liggitt mentioned this pull request Aug 17, 2020

ci-cloud-provider-gcp-conformance-latest failing with mismatched test / server version kubernetes/kubernetes#94051

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add periodic conformance test run against cloud-provider-gcp using kubetest2 #18714

Add periodic conformance test run against cloud-provider-gcp using kubetest2 #18714

michaelmdresser commented Aug 6, 2020 •

edited

Loading

michaelmdresser Aug 6, 2020

amwat Aug 6, 2020

michaelmdresser Aug 6, 2020

amwat Aug 6, 2020

michaelmdresser Aug 6, 2020

amwat Aug 6, 2020

michaelmdresser Aug 6, 2020

amwat left a comment

amwat Aug 6, 2020

amwat Aug 6, 2020

amwat Aug 6, 2020

amwat Aug 6, 2020

michaelmdresser Aug 6, 2020

amwat Aug 6, 2020

michaelmdresser Aug 7, 2020

amwat Aug 7, 2020

michaelmdresser Aug 7, 2020

amwat Aug 7, 2020

michaelmdresser Aug 7, 2020

amwat Aug 6, 2020

michaelmdresser Aug 6, 2020

amwat left a comment

amwat commented Aug 7, 2020

amwat commented Aug 7, 2020

spiffxp Aug 7, 2020

amwat commented Aug 7, 2020

spiffxp left a comment

k8s-ci-robot commented Aug 7, 2020

k8s-ci-robot commented Aug 7, 2020

liggitt commented Aug 17, 2020 •

edited

Loading

liggitt commented Aug 17, 2020

	extra_refs:
	- org: kubernetes-sigs
	repo: kind
	base_ref: master
	path_alias: sigs.k8s.io/kind

Add periodic conformance test run against cloud-provider-gcp using kubetest2 #18714

Add periodic conformance test run against cloud-provider-gcp using kubetest2 #18714

Conversation

michaelmdresser commented Aug 6, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amwat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amwat left a comment

Choose a reason for hiding this comment

amwat commented Aug 7, 2020

amwat commented Aug 7, 2020

Choose a reason for hiding this comment

amwat commented Aug 7, 2020

spiffxp left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Aug 7, 2020

k8s-ci-robot commented Aug 7, 2020

liggitt commented Aug 17, 2020 • edited Loading

liggitt commented Aug 17, 2020

michaelmdresser commented Aug 6, 2020 •

edited

Loading

liggitt commented Aug 17, 2020 •

edited

Loading