Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1.17 Fix: Revert reflector changes of PR #83520 to fix #86483 #86791 #86824

Merged

Conversation

@jpbetz
Copy link
Contributor

jpbetz commented Jan 3, 2020

What type of PR is this?

/kind bug

What this PR does / why we need it:

#83520 caused a regression in v1.17.0. Reflector relists are causing master rolling upgrade to fail for large clusters due to excessive list calls to etcd. See #86483.

Per #86430 (comment), we're reverting the reflector changes of #83520 in 1.17.

Which issue(s) this PR fixes:

Fixes #86483

Special notes for your reviewer:

This rolls back only the reflector changes. The server side changes were already tested for backward compatibility with old reflectors and are safe.

The pager change is left in place since they are unrelated to the regression. Note that #85272 is related and should be considered when reviewing.

Does this PR introduce a user-facing change?:

Fix v1.17.0 regression in reflector relist causing master rolling upgrade to fail for large clusters due to excessive list calls to etcd

/cc @wojtek-t @liggitt @smarterclayton

/sig api-machinery

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Jan 3, 2020

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.


Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

2 similar comments
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Jan 3, 2020

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.


Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Jan 3, 2020

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.


Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@jpbetz jpbetz force-pushed the jpbetz:revert-reflector-changes branch from 5e071c1 to a1ba57c Jan 3, 2020
@liggitt

This comment has been minimized.

Copy link
Member

liggitt commented Jan 5, 2020

/retest
/lgtm
/approve

@liggitt

This comment has been minimized.

Copy link
Member

liggitt commented Jan 5, 2020

/priority critical-urgent
/kind bug

Once this is merged, we will rerun the scale HA upgrade test scenario on the 1.17 branch (this was already run with this fix to verify it resolved the bug, but will be verified post-merge as well)

@liggitt

This comment has been minimized.

Copy link
Member

liggitt commented Jan 5, 2020

cc @kubernetes/patch-release-team for plans and timing of manual test post merge

@justaugustus

This comment has been minimized.

Copy link
Member

justaugustus commented Jan 6, 2020

Patch Release Team approval:
/lgtm
/approve

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Jan 6, 2020

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jpbetz, justaugustus, liggitt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit 23d323b into kubernetes:release-1.17 Jan 6, 2020
13 checks passed
13 checks passed
cla/linuxfoundation jpbetz authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-dependencies Job succeeded.
Details
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
tide In merge pool.
Details
@wojtek-t

This comment has been minimized.

Copy link
Member

wojtek-t commented Jan 7, 2020

@mm4tt will run the final tests to make a final confirmation for the release branch

@mm4tt

This comment has been minimized.

Copy link
Contributor

mm4tt commented Jan 7, 2020

I ran the test from at kubernetes:release-1.17 head today and can confirm that revert fixed the problem.

If someone is interested the grafana for the test can be found here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.