Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve node bootstrap time with node-authorizer and protokube in s3 #4

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

akursell-wish
Copy link

When using node-authorizer and a protokube image that must be pulled via http, node bootstrap takes more than 5 minutes because the order of dependencies is screwed up. What happens is:

  • kops starts protokube, node-authorizer, and kubelet at the same time
  • protokube cannot start until the protokube image is pulled, node-authorizer fails because protokube has not added the master ips to /etc/hosts, and kubelet fails because node-authorizer hasn't completed its task
  • After a minute or two, kops notices the failure, finally starts the LoadImage.protokube task that loads protokube, and restarts node-authorizer
  • node-authorizer fails again because protokube is still being pulled
  • protokube is pulled and is started
  • node-authorizer, after another minute or two, is restarted again by kops and succeeds
  • kubelet finally succeeds and the node joins the cluster

With this patch:

  • The protokube image is loaded first, before the protokube, node-authorizer, or kubelet services are
  • node-authorizer continuously retries until it succeeds

Node bootstrap now takes ~2 minutes.

I have tested this patch, and it works at least in our setup, but I don't know if this is unsound in other configurations. In particular, I noticed that kops seemed to depend on node-authorizer either failing or succeeding to make progress. If we didn't load the protokube image before starting node-authorizer, kops would hang indefinitely and never bootstrap.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
1 participant