Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Flaky test] gce-master-scale-correctness #115360

Closed
gracenng opened this issue Jan 27, 2023 · 11 comments · Fixed by #115697
Closed

[Flaky test] gce-master-scale-correctness #115360

gracenng opened this issue Jan 27, 2023 · 11 comments · Fixed by #115697
Assignees
Labels
kind/flake Categorizes issue or PR as related to a flaky test. sig/network Categorizes an issue or PR as relevant to SIG Network. triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@gracenng
Copy link
Member

gracenng commented Jan 27, 2023

Which jobs are failing flaky?

master-informing
gce-master-scale-correctness

Which tests are failing?

Kubernetes e2e suite: [It] [sig-network] LoadBalancers should not have connectivity disruption during rolling update with externalTrafficPolicy=Cluster

Since when has it been failing?

01-26 04:01 PST

Testgrid link

https://k8s-testgrid.appspot.com/sig-release-master-informing#gce-master-scale-correctness

Reason for failure (if possible)

{ failed [FAILED] error waiting for daemon pods to be ready: timed out waiting for the condition
In [It] at: test/e2e/network/loadbalancer.go:1865 @ 01/27/23 13:45:59.532
}

Anything else we need to know?

No response

Relevant SIG(s)

/sig-scalability

cc @Laura

@gracenng gracenng added the kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. label Jan 27, 2023
@k8s-ci-robot k8s-ci-robot added needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 27, 2023
@aojea
Copy link
Member

aojea commented Jan 27, 2023

/assign
/cc @andrewsykim

I warned this test was going to flake ;)

@aojea
Copy link
Member

aojea commented Jan 27, 2023

/sig network

@k8s-ci-robot k8s-ci-robot added sig/network Categorizes an issue or PR as relevant to SIG Network. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jan 27, 2023
@gracenng
Copy link
Member Author

Currently flaky so I updated the title but will keep issue open until further stability

@gracenng gracenng changed the title [Failing test] gce-master-scale-correctness [Flaky test] gce-master-scale-correctness Jan 28, 2023
@aojea
Copy link
Member

aojea commented Jan 28, 2023

yes, agreed

@aojea
Copy link
Member

aojea commented Jan 30, 2023

one failure is a GCE problem
https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/1618579963147980800

{ failed [FAILED] timed out waiting for service "test-lb-rolling-update" to have a load balancer: timed out waiting for the condition

the other a deployment problem
https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/1618942354264166400

{ failed [FAILED] error waiting for daemon pods to be ready: timed out waiting for the condition
In [It] at: test/e2e/network/loadbalancer.go:1865 @ 01/27/23 13:45:59.532

Failures look environmental, but let's keep it open until we get 5/5 greens on this test, there may be a symptom of a regression somewhere else

@shaneutt
Copy link
Member

shaneutt commented Feb 2, 2023

As per discussion of this issue in the SIG Network meeting today, marking this one as accepted.

/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Feb 2, 2023
@k8s-ci-robot
Copy link
Contributor

@drewhagen: The label(s) /label kind/flake cannot be applied. These labels are supported: api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactor, official-cve-feed. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

In response to this:

/label kind/flake

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot
Copy link
Contributor

@drewhagen: The label(s) /label flake cannot be applied. These labels are supported: api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactor, official-cve-feed. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

In response to this:

/label flake

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@drewhagen
Copy link
Member

/kind flake

@k8s-ci-robot k8s-ci-robot added the kind/flake Categorizes issue or PR as related to a flaky test. label Feb 9, 2023
@drewhagen
Copy link
Member

/remove-kind failing-test

@k8s-ci-robot k8s-ci-robot removed the kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. label Feb 9, 2023
@aojea
Copy link
Member

aojea commented Feb 11, 2023

/assign

it fails with

{ failed [FAILED] error waiting for daemon pods to start: timed out waiting for the condition

we need to cap it for large clusters

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/flake Categorizes issue or PR as related to a flaky test. sig/network Categorizes an issue or PR as relevant to SIG Network. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
Development

Successfully merging a pull request may close this issue.

6 participants
@drewhagen @shaneutt @aojea @k8s-ci-robot @gracenng and others