Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump dynamiclistener to fix apiserver outage issue #2833

Merged
merged 1 commit into from
May 3, 2022

Conversation

brandond
Copy link
Contributor

@brandond brandond commented Apr 30, 2022

Proposed Changes

Bump dynamiclistener to fix an issue where circular outages can prevent the supervisor from accepting new connections:

  1. supervisor on apiserver-only node tries to talk to bootstrap node during startup to reconcile bootstrap data (can't do it locally, no etcd)
  2. when accepting the TLS connection from the apiserver-only node, the etcd-only bootstrap node tries to talk to an apiserver to update the dynamiclistener certificate secret
  3. apiserver on apiserver-only node is still running in static pod from previous startup, but it is not responsive because it can't talk to any etcd servers through load-balancer tunnel, since it doesn't come up until step 1 is done
  4. apiserver-only rke2 supervisor startup fails due to TLS handshake eventually timing out

Running rke2-killall.sh on the apiserver-only node allows the start to succeed, as it kills the apiserver static pod, which in turn causes the dynamiclistener certificate update to fail (which IS handled properly) instead of just hanging.

Types of Changes

bugfix / version bump

Verification

See linked issue

Linked Issues

Further Comments

@codecov-commenter
Copy link

codecov-commenter commented Apr 30, 2022

Codecov Report

Merging #2833 (70acd7c) into master (84f5f67) will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master   #2833   +/-   ##
======================================
  Coverage    5.13%   5.13%           
======================================
  Files          24      24           
  Lines        1986    1986           
======================================
  Hits          102     102           
  Misses       1858    1858           
  Partials       26      26           
Flag Coverage Δ
unittests 5.13% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.


Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 84f5f67...70acd7c. Read the comment docs.

@thedadams
Copy link
Contributor

For reference, the created issue is here.

@brandond brandond marked this pull request as ready for review May 2, 2022 18:44
@brandond brandond requested a review from a team as a code owner May 2, 2022 18:44
Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
@brandond brandond merged commit 205668e into rancher:master May 3, 2022
@brandond brandond deleted the bump_dynamiclistener branch June 6, 2024 23:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants