Bug 1796147: pkg/server: serve config only to master in bootstrap server #1421
Conversation
The new cluster etcd operator flow is: 1) start bootstrap mcs 2) start etcd on bootstrap 3) wait for bootstrapping to finish i.e. atleast one control-plane is ready and there is MCS running on cluster 4) turn down bootstrap mcs What the above does is giving a chance to workers to grab the ignition config from the bootstap server which now stays up longer. However, by the time they attempt to create a CSR the kube-apiserver has rotated that bootstrap chain of trust out which causes the workers to error out with: Jan 29 19:55:20 ip-10-0-130-205 hyperkube[2623]: E0129 19:55:20.869251 2623 certificate_manager.go:421] Failed while requesting a signed certificate from the master: cannot create certificate signing request: Unauthorized The above results in workers not being able to join the cluster eventually. What this patch does is denying serving the configuration to all pools but master within the bootstrap server, effectively delaying workers to grab the wrong config from the wrong server. Workers will keep polling for configuration and they'll eventually grab the correct one from the server running within the new cluster. Signed-off-by: Antonio Murdaca <runcom@linux.com>
openshift-ci-robot
added
the
bugzilla/valid-bug
|
@runcom: This pull request references Bugzilla bug 1796147, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
openshift-ci-robot
added
the
size/M
openshift-ci-robot
added
the
approved
|
cc: @crawford |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: crawford, runcom The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
It's not terribly surprising that e2e-gcp-upgrade failed since it's using the bootstrap process prior to this change. /override ci/prow/e2e-gcp-upgrade |
|
@crawford: Overrode contexts on behalf of crawford: ci/prow/e2e-gcp-upgrade In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
@runcom: All pull requests linked via external trackers have merged. Bugzilla bug 1796147 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
@runcom: The following tests failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
/cherrypick fcos |
|
@vrutkovs: new pull request created: #1423 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
The new cluster etcd operator flow is:
What the above does is giving a chance to workers to grab
the ignition config from the bootstap server which now stays up longer.
However, by the time they attempt to create a CSR the kube-apiserver has
rotated that bootstrap chain of trust out which causes the workers to error out with:
Jan 29 19:55:20 ip-10-0-130-205 hyperkube[2623]: E0129 19:55:20.869251 2623 certificate_manager.go:421] Failed while requesting a signed certificate from the master: cannot create certificate signing request: Unauthorized
The above results in workers not being able to join the cluster eventually.
What this patch does is denying serving the configuration to all pools but master
within the bootstrap server, effectively delaying workers to grab the wrong config
from the wrong server. Workers will keep polling for configuration and they'll
eventually grab the correct one from the server running within the new cluster.
Signed-off-by: Antonio Murdaca runcom@linux.com