Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delay service readiness until after startuphooks have finished #5649

Merged
merged 9 commits into from Jun 15, 2022

Conversation

dereknola
Copy link
Contributor

@dereknola dereknola commented Jun 6, 2022

Proposed Changes

This PR introduces 3 changes

  1. Moves the startup hooks waitgroup into the runtime struct and has the server delay readiness until that wait group finishes
  2. Changes the "default" notifiee for systemd/openrc from the agent to the server.
  3. Adds a 1 second delay before responding back to secrets-encrypt requests. ETCD needs time to propagate changes from memory to disk.

This has the net result of delaying the return of calls to systemctl restart k3s until after the startup hooks have finished and k3s is in a more "ready" state.

Types of Changes

Bugfixes

Verification

  • Deploy a cluster with 1 server and 1 agent
  • After startup, restart the k3s-server service
  • Cluster should restart with no issues.

Linked Issues

rancher/rke2#897
#5442

User-Facing Change


Further Comments

@dereknola dereknola requested a review from a team as a code owner June 6, 2022 22:22
Copy link
Contributor

@brandond brandond left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one question about unsetting the variable in the agent code, LGTM otherwise.

pkg/agent/run.go Outdated Show resolved Hide resolved
…systemd

Signed-off-by: Derek Nola <derek.nola@suse.com>
Signed-off-by: Derek Nola <derek.nola@suse.com>
Signed-off-by: Derek Nola <derek.nola@suse.com>
Signed-off-by: Derek Nola <derek.nola@suse.com>
Signed-off-by: Derek Nola <derek.nola@suse.com>
Copy link
Contributor

@brandond brandond left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might add your fork as a replacement for k3s-io/k3s in RKE2, and do a quick build to make sure this works over there as well.

pkg/agent/run.go Outdated Show resolved Hide resolved
@dereknola
Copy link
Contributor Author

I've already done this. I have a RKE2 PR waiting in the wings for this PR to merge.

Signed-off-by: Derek Nola <derek.nola@suse.com>
Signed-off-by: Derek Nola <derek.nola@suse.com>
Signed-off-by: Derek Nola <derek.nola@suse.com>
pkg/server/secrets-encrypt.go Outdated Show resolved Hide resolved
Signed-off-by: Derek Nola <derek.nola@suse.com>
@dereknola dereknola merged commit a9b5a19 into k3s-io:master Jun 15, 2022
dereknola added a commit to dereknola/k3s that referenced this pull request Jun 15, 2022
…o#5649)

* Move startup hooks wg into a runtime pointer, check before notifying systemd
* Switch default systemd notification to server
* Add 1 sec delay to allow etcd to write to disk
Signed-off-by: Derek Nola <derek.nola@suse.com>
dereknola added a commit to dereknola/k3s that referenced this pull request Jun 15, 2022
…o#5649)

* Move startup hooks wg into a runtime pointer, check before notifying systemd
* Switch default systemd notification to server
* Add 1 sec delay to allow etcd to write to disk
Signed-off-by: Derek Nola <derek.nola@suse.com>
dereknola added a commit to dereknola/k3s that referenced this pull request Jun 15, 2022
…o#5649)

* Move startup hooks wg into a runtime pointer, check before notifying systemd
* Switch default systemd notification to server
* Add 1 sec delay to allow etcd to write to disk
Signed-off-by: Derek Nola <derek.nola@suse.com>
dereknola added a commit that referenced this pull request Jun 15, 2022
#5723)

* Move startup hooks wg into a runtime pointer, check before notifying systemd
* Switch default systemd notification to server
* Add 1 sec delay to allow etcd to write to disk
Signed-off-by: Derek Nola <derek.nola@suse.com>
@dereknola dereknola deleted the startuphooks_wait branch June 28, 2022 20:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants