Document the workaround for kube-proxy's mishandled "hostname-override" flag #9878

seh · 2018-08-16T16:36:20Z

Following up from @dlipovetsky's earlier #9663, with all credit due to him for the core idea, offer a shell script workaround for kube-proxy when run on machines whose hostname must differ from the node name, such as when running in AWS with the corresponding cloud provider enabled. kubernetes/kubeadm#857 captures the problem in more detail.

See here for preceding discussion of this approach, refined here slightly.

k8s-ci-robot · 2018-08-16T16:36:28Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: bradamant3

If they are not already assigned, you can assign the PR to them by writing /assign @bradamant3 in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8sio-netlify-preview-bot · 2018-08-16T16:40:55Z

Deploy preview for kubernetes-io-master-staging ready!

Built with commit ea74837

https://deploy-preview-9878--kubernetes-io-master-staging.netlify.com

detiber · 2018-08-17T14:28:34Z

/lgtm
@dlipovetsky ptal
@Bradamant3 could you provide a review?

dlipovetsky · 2018-08-17T17:45:25Z

This looks good. Thanks for adding the explanatory notes @seh. I do find the JSON patch to be easier to understand, but I think that's no big deal 😄.

seh · 2018-08-17T17:48:51Z

content/en/docs/setup/independent/troubleshooting-kubeadm.md

@@ -201,41 +200,66 @@ Error from server: Get https://10.19.0.41:10250/containerLogs/default/mysql-ddc6

 ## Services with externalTrafficPolicy=Local are not reachable

-On nodes where the hostname for the kubelet is overridden using the `--hostname-override` option, kube-proxy will default to treating 127.0.0.1 as the node IP, which results in rejecting connections for Services configured for `externalTrafficPolicy=Local`. This situation can be verified by checking the output of `kubectl -n kube-system logs <kube-proxy pod name>`:
+On machines where the node name for the kubelet is overridden using the `--hostname-override` option, such that the node name no longer matches the machine's hostname, kube-proxy will default to treating 127.0.0.1 as the node's IP address, which results in  rejecting connections for Services configured for `externalTrafficPolicy=Local`. This situation can be verified by checking the output of `kubectl -n kube-system logs <kube-proxy pod name>`:


There's an extra space between "in" and "rejecting." I can push an update to fix that.

At present kube-proxy does not honor its --hostname-override flag when used in concert with its --config flag; though it reads the former flag value, it overrides its with values read from the specified configuration file. In this case that file's "hostnameOverride" field's value takes precedence, even when absent, yielding an empty value. Replace the previous documented workaround with one that customizes a copy of the configuration file acquired from the ConfigMap.

Capitalize each item as a sentence, and restore intended hanging indentation of the cgroup driver-related item.

detiber · 2018-08-17T17:52:56Z

/lgtm

zacharysarah

@seh Great start! ✨ Some feedback for docs quality, otherwise LGTM.

zacharysarah · 2018-10-01T23:12:15Z

content/en/docs/setup/independent/troubleshooting-kubeadm.md

-  
- 1. Install docker again following instructions
+
+ 1. Install Docker again following instructions


Change this paragraph to:

1. Re-install [Docker](/docs/setup/independent/install-kubeadm/#installing-docker).

So:

Re-install Docker.

That's a fine suggestion, here and in what follows, but note that I was just trying to adjust the formatting and capitalization here. This is the unfortunate phenomenon of drawing focus only to what one touches, at the expense of all the other parts that were ignored.

Still, I appreciate that you did think through your suggestions, so I'll take them into account when I can make another editing pass here.

zacharysarah · 2018-10-01T23:13:41Z

content/en/docs/setup/independent/troubleshooting-kubeadm.md

-    [Configure cgroup driver used by kubelet on Master Node](/docs/setup/independent/install-kubeadm/#configure-cgroup-driver-used-by-kubelet-on-master-node)
-    for detailed instructions.
+ 1. Change the kubelet config to match the Docker cgroup driver manually.  
+    You can refer to [Configure cgroup driver used by kubelet on Master Node](/docs/setup/independent/install-kubeadm/#configure-cgroup-driver-used-by-kubelet-on-master-node) for detailed instructions.


Change to:

See "[Configure cgroup driver used by kubelet on Master Node](/docs/setup/independent/install-kubeadm/#configure-cgroup-driver-used-by-kubelet-on-master-node)" for detailed instructions.

So:

See "Configure cgroup driver used by kubelet on Master Node" for detailed instructions.

zacharysarah · 2018-10-01T23:14:31Z

content/en/docs/setup/independent/troubleshooting-kubeadm.md


- control plane Docker containers are crashlooping or hanging. You can check this by running `docker ps` and investigating each container by running `docker logs`.
+- Control plane Docker containers are crashlooping or hanging.  


Change to:

- Control plane Docker containers crash or hang.

zacharysarah · 2018-10-01T23:17:44Z

content/en/docs/setup/independent/troubleshooting-kubeadm.md

@@ -201,41 +200,65 @@ Error from server: Get https://10.19.0.41:10250/containerLogs/default/mysql-ddc6

 ## Services with externalTrafficPolicy=Local are not reachable

-On nodes where the hostname for the kubelet is overridden using the `--hostname-override` option, kube-proxy will default to treating 127.0.0.1 as the node IP, which results in rejecting connections for Services configured for `externalTrafficPolicy=Local`. This situation can be verified by checking the output of `kubectl -n kube-system logs <kube-proxy pod name>`:
+On machines where the node name for the kubelet is overridden using the `--hostname-override` option, such that the node name no longer matches the machine's hostname, kube-proxy will default to treating 127.0.0.1 as the node's IP address, which results in rejecting connections for Services configured for `externalTrafficPolicy=Local`. This situation can be verified by checking the output of `kubectl -n kube-system logs <kube-proxy pod name>`:


Replace with:

On machines where the `--hostname-override` option overrides the node name so that the kubelet no longer matches the machine's hostname, kube-proxy defaults to treating 127.0.0.1 as the node's IP address, which results in rejecting connections for Services configured for `externalTrafficPolicy=Local`. You can verify this situation by checking the output of `kubectl -n kube-system logs <kube-proxy pod name>`:

So:

On machines where the --hostname-override option overrides the node name so that the kubelet no longer matches the machine's hostname, kube-proxy defaults to treating 127.0.0.1 as the node's IP address, which results in rejecting connections for Services configured for externalTrafficPolicy=Local. You can verify this situation by checking the output of kubectl -n kube-system logs <kube-proxy pod name>:

zacharysarah · 2018-10-01T23:23:11Z

content/en/docs/setup/independent/troubleshooting-kubeadm.md


 ```sh
 W0507 22:33:10.372369       1 server.go:586] Failed to retrieve node info: nodes "ip-10-0-23-78" not found
 W0507 22:33:10.372474       1 proxier.go:463] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
 ```

-A workaround for this is to modify the kube-proxy DaemonSet in the following way:
+Just as the kubelet accepts a `--hostname-override` flag to use a different name for the node, so too does kube-proxy. Unfortunately, though, due to an unresolved defect, kube-proxy does not honor that flag's value when used in combination with the `--config` flag specifying a configuration file. Hence we can't use kube-proxy's `--hostname-override` flag to tell it the proper name of our node.


Replace with:

{{< caution >}} **Caution:** Do not use kube-proxy's `--hostname-override` flag to indicate the name of a node. While kube-proxy also accepts a `--hostname-override` flag, an unresolved defect causes kube-proxy not to honor that flag's value when used in combination with the `--config` flag specifying a configuration file. {{< /caution >}}

So:

Caution: Do not use kube-proxy's --hostname-override flag to indicate the name of a node.

While kube-proxy also accepts a --hostname-override flag, an unresolved defect causes kube-proxy not to honor that flag's value when used in combination with the --config flag specifying a configuration file.

stevesloka · 2018-10-26T20:37:36Z

@seh I got a PR merged which fixes this problem in kube-proxy (kubernetes/kubernetes#69340). I went to update docs and found your PR was already in progress. Just wanted to update you on this which affects the content of this PR.

seh · 2018-11-18T20:34:25Z

@seh I got a PR merged which fixes this problem in kube-proxy (kubernetes/kubernetes#69340).

Almost a month later, @stevesloka, I finally had the opportunity to put your patch to use this morning, and it worked! Thank you for the fix.

I still had to patch the DaemonSet for kube-proxy that kubeadm creates, but at least it's less messy alteration. @anguslees is on to something with his suggestion here.

mdlinville · 2018-12-11T22:22:03Z

We'd love to accept this, but it has some pending feedback and needs to be rebased to resolve conflicts. Thanks!

/hold

seh · 2018-12-14T03:08:49Z

@mistyhacks, as of version 1.13.0, this workaround is no longer necessary. It was pertinent for a few months before then, but not now. I suggest we abandon the change.

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 16, 2018

k8s-ci-robot requested review from Rajakavitha1 and zhangxiaoyu-zidif August 16, 2018 16:36

seh mentioned this pull request Aug 16, 2018

Verified workaround for kube-proxy hostnameOverride issue #9663

Closed

k8s-ci-robot assigned detiber Aug 17, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 17, 2018

seh commented Aug 17, 2018

View reviewed changes

Steven E. Harris added 2 commits August 17, 2018 13:50

Copyedit blocking-on-control-plane section

ea74837

Capitalize each item as a sentence, and restore intended hanging indentation of the cgroup driver-related item.

seh force-pushed the doc-workaround-for-kube-proxy-hostname branch from e601116 to ea74837 Compare August 17, 2018 17:51

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 17, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 17, 2018

zacharysarah suggested changes Oct 1, 2018

View reviewed changes

zacharysarah added the Docs Review: Open Issues label Oct 1, 2018

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 11, 2018

mdlinville closed this Dec 14, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document the workaround for kube-proxy's mishandled "hostname-override" flag #9878

Document the workaround for kube-proxy's mishandled "hostname-override" flag #9878

seh commented Aug 16, 2018

k8s-ci-robot commented Aug 16, 2018

k8sio-netlify-preview-bot commented Aug 16, 2018 •

edited

detiber commented Aug 17, 2018

dlipovetsky commented Aug 17, 2018

seh Aug 17, 2018

detiber commented Aug 17, 2018

zacharysarah left a comment

zacharysarah Oct 1, 2018

seh Nov 11, 2018

zacharysarah Oct 1, 2018

zacharysarah Oct 1, 2018

zacharysarah Oct 1, 2018

zacharysarah Oct 1, 2018

stevesloka commented Oct 26, 2018

seh commented Nov 18, 2018

mdlinville commented Dec 11, 2018

seh commented Dec 14, 2018


		1. Install docker again following instructions

		1. Install Docker again following instructions


		- control plane Docker containers are crashlooping or hanging. You can check this by running `docker ps` and investigating each container by running `docker logs`.
		- Control plane Docker containers are crashlooping or hanging.

Document the workaround for kube-proxy's mishandled "hostname-override" flag #9878

Document the workaround for kube-proxy's mishandled "hostname-override" flag #9878

Conversation

seh commented Aug 16, 2018

k8s-ci-robot commented Aug 16, 2018

k8sio-netlify-preview-bot commented Aug 16, 2018 • edited

detiber commented Aug 17, 2018

dlipovetsky commented Aug 17, 2018

seh Aug 17, 2018

Choose a reason for hiding this comment

detiber commented Aug 17, 2018

zacharysarah left a comment

Choose a reason for hiding this comment

zacharysarah Oct 1, 2018

Choose a reason for hiding this comment

seh Nov 11, 2018

Choose a reason for hiding this comment

zacharysarah Oct 1, 2018

Choose a reason for hiding this comment

zacharysarah Oct 1, 2018

Choose a reason for hiding this comment

zacharysarah Oct 1, 2018

Choose a reason for hiding this comment

zacharysarah Oct 1, 2018

Choose a reason for hiding this comment

stevesloka commented Oct 26, 2018

seh commented Nov 18, 2018

mdlinville commented Dec 11, 2018

seh commented Dec 14, 2018

k8sio-netlify-preview-bot commented Aug 16, 2018 •

edited