Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix rollout of the discovery service #190

Merged
merged 1 commit into from
May 25, 2023

Conversation

roivaz
Copy link
Member

@roivaz roivaz commented May 25, 2023

With the addition of health checks, the rollout of new versions of the discovery service is slower because the new pod takees some seconds to pass the health checks. This makes that both the old controllers and the new ones run in parallel for some seconds, trying to write to the same EnvoyConfigRevisions and causing problems. To fix this:

  • Change the Deployment rollout strategy to "replace"
  • Add controller lock
  • Shutdown the xDS sever faster, without graceful shutdown

I have tested an upgrade from v0.11.1 to a version with the changes in this PR and it fixes the problems.

/kind bug
/priority critical-urgent
/assign
/ok-to-test

@3scale-robot 3scale-robot added kind/bug Categorizes issue or PR as related to a bug. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. needs-size Indicates a PR or issue lacks a `size/foo` label and requires one. labels May 25, 2023
@3scale-robot 3scale-robot added size/S Requires less than a day to complete the PR or the issue. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-size Indicates a PR or issue lacks a `size/foo` label and requires one. labels May 25, 2023
@slopezz
Copy link
Member

slopezz commented May 25, 2023

/lgtm

@3scale-robot 3scale-robot added the lgtm Indicates that a PR is ready to be merged. label May 25, 2023
@3scale-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: b56536786f6f5a7393b3ac1d70f243a272a0f680

@raelga
Copy link
Contributor

raelga commented May 25, 2023

/lgtm

@roivaz
Copy link
Member Author

roivaz commented May 25, 2023

/approve

@3scale-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: roivaz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@3scale-robot 3scale-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 25, 2023
@3scale-robot 3scale-robot merged commit bea9b76 into main May 25, 2023
@3scale-robot 3scale-robot deleted the fix-discoveryservice-rollout branch May 25, 2023 14:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. kind/bug Categorizes issue or PR as related to a bug. lgtm Indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. size/S Requires less than a day to complete the PR or the issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants