Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stop using ginkgo.flakeAttempts in e2e jobs #15516

Merged
merged 2 commits into from Dec 14, 2019

Conversation

@spiffxp
Copy link
Member

spiffxp commented Dec 6, 2019

This is a proof of concept of removing flake attempts from all jobs per kubernetes/kubernetes#68091 (comment)

See individual commits for which jobs are impacted by which change

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Dec 6, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: spiffxp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

spiffxp added 2 commits Dec 6, 2019
Anyplace that used 2 was retrying when failing a test, which we have
determined doesn't seem to provide any significantly better pass rate
these days.

Anyplace that used 1 was using the default value, so doesn't need to
explicitly set it

Affects:
- ci-cadvisor-e2e
- ci-cloud-provider-azure-conformance
- ci-cloud-provider-azure-conformance-vmss
- ci-cloud-provider-azure-multiple-zones
- ci-cloud-provider-azure-serial
- ci-cloud-provider-azure-slow
- ci-cloud-provider-azure-slow-vmss
- ci-containerd-node-e2e
- ci-containerd-node-e2e-1-2
- ci-containerd-node-e2e-1-3
- ci-containerd-node-e2e-features
- ci-containerd-node-e2e-features-1-2
- ci-containerd-node-e2e-features-1-3
- ci-cos-containerd-node-e2e
- ci-cos-containerd-node-e2e-features
- ci-cri-containerd-node-e2e
- ci-cri-containerd-node-e2e-features
- ci-kubernetes-e2e-aks-engine-azure-1-14-windows
- ci-kubernetes-e2e-aks-engine-azure-1-15-windows
- ci-kubernetes-e2e-aks-engine-azure-1-16-windows
- ci-kubernetes-e2e-aks-engine-azure-1-17-windows
- ci-kubernetes-e2e-aks-engine-azure-master-staging-windows
- ci-kubernetes-e2e-aks-engine-azure-master-windows
- ci-kubernetes-e2e-aws-eks-1-13-conformance
- ci-kubernetes-e2e-aws-eks-1-13-correctness
- ci-kubernetes-e2e-gce-private-cluster-correctness
- ci-kubernetes-e2e-gce-scale-correctness
- ci-kubernetes-e2e-kops-aws
- ci-kubernetes-e2e-kops-aws-beta
- ci-kubernetes-e2e-kops-aws-canary
- ci-kubernetes-e2e-kops-aws-channelalpha
- ci-kubernetes-e2e-kops-aws-ena-nvme
- ci-kubernetes-e2e-kops-aws-ha-uswest2
- ci-kubernetes-e2e-kops-aws-imagecentos7
- ci-kubernetes-e2e-kops-aws-imageubuntu1604
- ci-kubernetes-e2e-kops-aws-networking-kopeio-vxlan
- ci-kubernetes-e2e-kops-aws-newrunner
- ci-kubernetes-e2e-kops-aws-sig-cli
- ci-kubernetes-e2e-kops-aws-stable1
- ci-kubernetes-e2e-kops-aws-stable2
- ci-kubernetes-e2e-kops-aws-stable3
- ci-kubernetes-e2e-kops-aws-updown
- ci-kubernetes-e2e-kops-aws-vpc-cni
- ci-kubernetes-e2e-kops-aws-weave
- ci-kubernetes-e2e-kops-gce
- ci-kubernetes-e2e-kops-gce-canary
- ci-kubernetes-e2e-kops-gce-channelalpha
- ci-kubernetes-e2e-kops-gce-ha
- pull-cadvisor-e2e
- pull-cloud-provider-azure-e2e
- pull-cri-containerd-node-e2e
- pull-kops-e2e-kubernetes-aws
- pull-kops-e2e-kubernetes-aws-1-15
- pull-kops-e2e-kubernetes-aws-1-16
- pull-kubernetes-e2e-aks-engine-azure-windows
- pull-kubernetes-e2e-gke
- pull-kubernetes-e2e-kops-aws
- pull-kubernetes-node-e2e
- pull-kubernetes-node-e2e-alpha
- pull-kubernetes-node-e2e-containerd
this plumbs into kubernetes/kubernetes/hack/ginkgo-e2e.sh to set
a FLAKE_ATTEMPTS env var to 2, which is passed to ginkgo via the
--ginkgo.flakeAttempts flag

Affects:
- pull-kubernetes-e2e-containerd-gce
- pull-kubernetes-e2e-gce
- pull-kubernetes-e2e-gce-alpha-features
- pull-kubernetes-e2e-gce-csi-serial
- pull-kubernetes-e2e-gce-device-plugin-gpu
- pull-kubernetes-e2e-gce-iscsi
- pull-kubernetes-e2e-gce-iscsi-serial
- pull-kubernetes-e2e-gce-rbe
- pull-kubernetes-e2e-gce-storage-disruptive
- pull-kubernetes-e2e-gce-storage-slow
- pull-kubernetes-e2e-gce-storage-slow-rbe
- pull-kubernetes-e2e-gce-storage-snapshot
- pull-kubernetes-e2e-gci-gce-autoscaling
- pull-release-cluster-up
@spiffxp

This comment has been minimized.

Copy link
Member Author

spiffxp commented Dec 10, 2019

/hold
merge deadline friday december 13th

@spiffxp spiffxp force-pushed the spiffxp:no-more-flake-attempts branch from 3acd492 to 0aa5d08 Dec 10, 2019
@spiffxp spiffxp changed the title [wip] stop using ginkgo.flakeAttempts in e2e jobs stop using ginkgo.flakeAttempts in e2e jobs Dec 10, 2019
@aojea

This comment has been minimized.

Copy link
Member

aojea commented Dec 10, 2019

/cc

@k8s-ci-robot k8s-ci-robot requested a review from aojea Dec 10, 2019
@spiffxp

This comment has been minimized.

Copy link
Member Author

spiffxp commented Dec 13, 2019

/hold cancel

@BenTheElder

This comment has been minimized.

Copy link
Member

BenTheElder commented Dec 14, 2019

/lgtm
Friday flake day

@k8s-ci-robot k8s-ci-robot merged commit 2aa69a4 into kubernetes:master Dec 14, 2019
4 of 5 checks passed
4 of 5 checks passed
tide Not mergeable. Retesting: pull-test-infra-bazel
Details
cla/linuxfoundation spiffxp authorized
Details
pull-test-infra-bazel Job succeeded.
Details
pull-test-infra-verify-file-perms Job succeeded.
Details
pull-test-infra-yamllint Job succeeded.
Details
Release Engineering automation moved this from In progress to Done (1.18) Dec 14, 2019
@k8s-ci-robot k8s-ci-robot added this to the v1.18 milestone Dec 14, 2019
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Dec 14, 2019

@spiffxp: Updated the job-config configmap in namespace default at cluster default using the following files:

  • key cadvisor.yaml using file config/jobs/cadvisor/cadvisor.yaml
  • key containerd-cri-presubmit-jobs.yaml using file config/jobs/containerd/cri/containerd-cri-presubmit-jobs.yaml
  • key cloud-provider-azure-config.yaml using file config/jobs/kubernetes-sigs/cloud-provider-azure/cloud-provider-azure-config.yaml
  • key sig-windows-config.yaml using file config/jobs/kubernetes-sigs/sig-windows/sig-windows-config.yaml
  • key kops-config.yaml using file config/jobs/kubernetes/kops/kops-config.yaml
  • key sig-cli-config.yaml using file config/jobs/kubernetes/sig-cli/sig-cli-config.yaml
  • key eks-periodics.yaml using file config/jobs/kubernetes/sig-cloud-provider/aws/eks/eks-periodics.yaml
  • key kops-periodics.yaml using file config/jobs/kubernetes/sig-cloud-provider/aws/kops/kops-periodics.yaml
  • key kops-presubmits.yaml using file config/jobs/kubernetes/sig-cloud-provider/aws/kops/kops-presubmits.yaml
  • key gcp-gce.yaml using file config/jobs/kubernetes/sig-cloud-provider/gcp/gcp-gce.yaml
  • key gcp-gke.yaml using file config/jobs/kubernetes/sig-cloud-provider/gcp/gcp-gke.yaml
  • key containerd.yaml using file config/jobs/kubernetes/sig-node/containerd.yaml
  • key sig-node-presubmit.yaml using file config/jobs/kubernetes/sig-node/sig-node-presubmit.yaml
  • key 1.14.yaml using file config/jobs/kubernetes/sig-release/release-branch-jobs/1.14.yaml
  • key 1.15.yaml using file config/jobs/kubernetes/sig-release/release-branch-jobs/1.15.yaml
  • key 1.16.yaml using file config/jobs/kubernetes/sig-release/release-branch-jobs/1.16.yaml
  • key 1.17.yaml using file config/jobs/kubernetes/sig-release/release-branch-jobs/1.17.yaml
  • key sig-scalability-experimental-periodic-jobs.yaml using file config/jobs/kubernetes/sig-scalability/sig-scalability-experimental-periodic-jobs.yaml
  • key sig-scalability-release-blocking-jobs.yaml using file config/jobs/kubernetes/sig-scalability/sig-scalability-release-blocking-jobs.yaml
  • key kubetest-canaries.yaml using file config/jobs/kubernetes/sig-testing/kubetest-canaries.yaml

In response to this:

This is a proof of concept of removing flake attempts from all jobs per kubernetes/kubernetes#68091 (comment)

See individual commits for which jobs are impacted by which change

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@dims

This comment has been minimized.

Copy link
Member

dims commented Dec 14, 2019

/lgtm

better late than never :)

@BenTheElder

This comment has been minimized.

Copy link
Member

BenTheElder commented Dec 14, 2019

so e2e-k8s.sh in kind is setting export GINKGO_TOLERATE_FLAKES="${GINKGO_TOLERATE_FLAKES:-y}" which maps to --ginkgo.flakeAttempts=2 inside Kubernetes' hack/ginkgo-e2e.sh

we can either change kind's default testing to not do this (filing that PR) or set GINKGO_TOLERATE_FLAKES here (which we already do in the conformance jobs)

AFAICT, we're trying to remove all usage of this, right?

@BenTheElder

This comment has been minimized.

@spiffxp spiffxp deleted the spiffxp:no-more-flake-attempts branch Dec 16, 2019
@spiffxp

This comment has been minimized.

Copy link
Member Author

spiffxp commented Dec 16, 2019

List of jobs impacted in commits, copy-pasted into kubernetes/kubernetes#68091 (comment) for reference

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Release Engineering
  
Done (1.18)
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants
You can’t perform that action at this time.