Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[release-1.21] Bump dynamiclistener to fix apiserver outage issue #2847

Merged

Conversation

brandond
Copy link
Contributor

@brandond brandond commented May 3, 2022

Proposed Changes

Bump dynamiclistener to fix an issue where circular outages can prevent the supervisor from accepting new connections:

  1. supervisor on apiserver-only node tries to talk to bootstrap node during startup to reconcile bootstrap data (can't do it locally, no etcd)
  2. when accepting the TLS connection from the apiserver-only node, the etcd-only bootstrap node tries to talk to an apiserver to update the dynamiclistener certificate secret
  3. apiserver on apiserver-only node is still running in static pod from previous startup, but it is not responsive because it can't talk to any etcd servers through load-balancer tunnel, since it doesn't come up until step 1 is done
  4. apiserver-only rke2 supervisor startup fails due to TLS handshake eventually timing out

Running rke2-killall.sh on the apiserver-only node allows the start to succeed, as it kills the apiserver static pod, which in turn causes the dynamiclistener certificate update to fail (which IS handled properly) instead of just hanging.

Types of Changes

bugfix / version bump

Verification

See linked issue

Linked Issues

Further Comments

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
@brandond brandond requested a review from a team as a code owner May 3, 2022 01:55
@brandond brandond merged commit 57ade3c into rancher:release-1.21 May 3, 2022
@brandond brandond deleted the bump_dynamiclistener_release-1.21 branch June 6, 2024 23:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants