Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing Test: [sig-network] Proxy version v1 should proxy to cadvisor using proxy subresource failing in sig-release-master-upgrade suite #64158

Closed
AishSundar opened this issue May 22, 2018 · 22 comments
Closed

Failing Test: [sig-network] Proxy version v1 should proxy to cadvisor using proxy subresource failing in sig-release-master-upgrade suite #64158

AishSundar opened this issue May 22, 2018 · 22 comments
Assignees
@krzyzacy @luxas @dixudx
Labels
kind/bug Categorizes issue or PR as related to a bug. kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. sig/network Categorizes an issue or PR as relevant to SIG Network. sig/node Categorizes an issue or PR as relevant to SIG Node.
Milestone
v1.11

Comments

@AishSundar
Copy link
Contributor

Failing Job

##Failing Test
[sig-network] Proxy version v1 should proxy to cadvisor using proxy subresource failing in sig-release-master-upgrade suite

Triage results

https://storage.googleapis.com/k8s-gubernator/triage/index.html?sig=network&job=ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel

This test started failing consistently in the upgrade jobs since 5/21 evening.

/kind bug
/priority failing-test
/priority important-soon
/sig network
/milestone v1.11
@kubernetes/sig-network-bugs

cc @jberkus @tpepper

/assign @bjhaid

@foxie @bjhaid we are seeing consistent network e2e tests failing in sig-release-master-upgrade jobs. Can you please triage this as priority and let us know if we can resolve it by Code flush (5/29)?
@k8s-ci-robot
Copy link
Contributor

@AishSundar: GitHub didn't allow me to assign the following users: bjhaid.

Note that only kubernetes members and repo collaborators can be assigned.

In response to this:

Failing Job

##Failing Test
[sig-network] Proxy version v1 should proxy to cadvisor using proxy subresource failing in sig-release-master-upgrade suite

Triage results

https://storage.googleapis.com/k8s-gubernator/triage/index.html?sig=network&job=ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel

This test started failing consistently in the upgrade jobs since 5/21 evening.

/kind bug
/priority failing-test
/priority important-soon
/sig network
/milestone v1.11
@kubernetes/sig-network-bugs

cc @jberkus @tpepper

/assign @bjhaid

@foxie @bjhaid we are seeing consistent network e2e tests failing in sig-release-master-upgrade jobs. Can you please triage this as priority and let us know if we can resolve it by Code flush (5/29)?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the sig/network Categorizes an issue or PR as relevant to SIG Network. label May 22, 2018
@k8s-ci-robot k8s-ci-robot added this to the v1.11 milestone May 22, 2018
@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/network Categorizes an issue or PR as relevant to SIG Network. labels May 22, 2018
@MrHohn
Copy link
Member

MrHohn commented May 22, 2018

Commit diff: af69af2...9eb0c35

Seems #63881 disabled the public cadvisor port by default. Guessing that is the cause?
@luxas

@AishSundar
Copy link
Contributor Author

@luxas should this failing test also be handled as part of your change?

@luxas
Copy link
Member

luxas commented May 22, 2018

The "fix" here is to just not run this e2e test against a v1.11-upgraded cluster. How do we disable running that against v1.11 infra in the GCE test-infra? cc @krousey

@AishSundar this is not a regression in terms of functionality, it's a test which is running although it shouldn't.

@AishSundar
Copy link
Contributor Author

/cc @BenTheElder

@BenTheElder
Copy link
Member

@luxas which suites run are in the job args, https://github.com/kubernetes/test-infra/blob/038ec489c8dfb0e5c57cfc667a22002cb8c8876f/jobs/config.json#L6214

@luxas
Copy link
Member

luxas commented May 22, 2018

@BenTheElder and just making ginkgo skip that e2e for these upgrade suites sounds plausible to you?

@luxas luxas self-assigned this May 22, 2018
@luxas luxas added sig/node Categorizes an issue or PR as relevant to SIG Node. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. labels May 22, 2018
@BenTheElder
Copy link
Member

@luxas, yeah ginkgo flags should be controlling which testcases actually run, and I think we've done this before. @krzyzacy should be able to confirm.

@krzyzacy
Copy link
Member

hummm, sounds like we should just remove this test?

@luxas
Copy link
Member

luxas commented May 22, 2018

@krzyzacy The e2e test in question was removed in v1.11, but it's still running apparently in v1.10-skewed upgraded clusters, which is the problem.

@krzyzacy
Copy link
Member

@liggitt I forgot how did we handle this in 1.10? I thought we also disabled the test in the past release to work around, instead of use ginkgo skip, maybe you can confirm.

@liggitt
Copy link
Member

liggitt commented May 23, 2018

like this: https://github.com/kubernetes/kubernetes/pull/61135/files#diff-e4236fde991a93836f485ce2d85ef8aa

@krzyzacy
Copy link
Member

ah, great, thanks!

@dixudx dixudx mentioned this issue May 23, 2018
Merged
k8s-github-robot pushed a commit that referenced this issue May 25, 2018
Merge pull request #64184 from dixudx/e2e_cadvisor_skew
e9491e9 
Automatic merge from submit-queue.

Prevent 1.10 e2es testing deprecated CAdvisorPort in 1.11

**What this PR does / why we need it**:
The public cadvisor port by default is disabled in #63881, targeted for v1.11.

But 1.10 e2e tests get run against v1.11+ masters during upgrade tests.

https://k8s-testgrid.appspot.com/sig-release-master-upgrade#gce-1.10-master-upgrade-cluster-parallel

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #64158

**Special notes for your reviewer**:
/cc luxas BenTheElder krzyzacy
 
**Release note**:

```release-note
Prevent 1.10 e2es testing deprecated CAdvisorPort in v1.11
```
@AishSundar
Copy link
Contributor Author

I see this test is still failing in the 2 jobs in the latest run (at commit #64400) which has the merged PR.

@krzyzacy can you PTAL?

@krzyzacy
Copy link
Member

@AishSundar can you link a build log?

@AishSundar
Copy link
Contributor Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel/3599?log#log

@krzyzacy
Copy link
Member

hummmm... @liggitt I'm a bit confused why #64184 doesn't work

@AishSundar
Copy link
Contributor Author

/status in-progress

@AishSundar
Copy link
Contributor Author

As per @liggitt this CL shows the right method for skipping. @krzyzacy we should probably make the same change for this issue as well.

@krzyzacy
Copy link
Member

/assign

@k8s-github-robot
Copy link

[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process

@AishSundar @dixudx @krzyzacy @luxas

Note: If this issue is not resolved or labeled as priority/critical-urgent by Tuesday, June 5th it will be moved out of the v1.11 milestone.

Issue Labels
  • sig/cluster-lifecycle sig/network sig/node: Issue will be escalated to these SIGs if needed.
  • priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts.
  • kind/bug: Fixes a bug discovered during the current release.
Help

@krzyzacy krzyzacy mentioned this issue May 30, 2018
Merged
k8s-github-robot pushed a commit that referenced this issue May 31, 2018
Merge pull request #64524 from krzyzacy/release-1.10
547d38b 
Automatic merge from submit-queue.

properly skip cadvisor proxy test

xref #64158

per #64158 (comment)

/assign @liggitt @AishSundar
@AishSundar
Copy link
Contributor Author

Tests are passing now !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@krzyzacy krzyzacy

@luxas luxas

@dixudx dixudx

Labels
kind/bug Categorizes issue or PR as related to a bug. kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. sig/network Categorizes an issue or PR as relevant to SIG Network. sig/node Categorizes an issue or PR as relevant to SIG Node.
Projects
None yet
Milestone
v1.11
Development

No branches or pull requests
9 participants
@BenTheElder @liggitt @krzyzacy @luxas @dixudx @MrHohn @k8s-github-robot @AishSundar @k8s-ci-robot