Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable control plane to maintain during reboot #467

Closed
4 tasks
zoetrope opened this issue May 27, 2021 · 0 comments
Closed
4 tasks

Enable control plane to maintain during reboot #467

zoetrope opened this issue May 27, 2021 · 0 comments

Comments

@zoetrope
Copy link
Member

zoetrope commented May 27, 2021

What

Maintenance operation does not run while reboot operation is running.

  • Endpoints are not updated if one control-plane is down.
  • controller-manager and scheduler do not recover if they crash.

How

Run reboot operation on another go routine

  • Modify rebootOp as follows
    • after rebooting, remove cke.cybozu.com/reboot annotation
    • wait until the node is schedulable
    • remove the rebooted node from the reboot queue
  • Modify rebootUncordonOp as follows
    • only uncordon for the node which is unschedulable and no cke.cybozu.com/reboot` annotation
  • Register records of the reboot operation and do the operation by using a goroutine in order to avoid record registration conflicts with other operations.
  • Disable sabakan integration for the certain period during a reboot operation in order to avoid the frequent master node change

Checklist

  • Finish implentation of the issue
  • Test all functions
  • Have enough logs to trace activities
  • Notify developers of necessary actions
@zoetrope zoetrope changed the title Allow control plane maintenance during reboot Enable control plane maintenance during reboot May 27, 2021
@zoetrope zoetrope changed the title Enable control plane maintenance during reboot Enable control plane to maintain during reboot May 27, 2021
@binoue binoue self-assigned this Jun 1, 2021
@binoue binoue removed their assignment Jun 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants