Disable the public cadvisor port by default #63881
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
k8s-ci-robot
added
release-note-action-required
luxas
added
priority/important-soon
k8s-github-robot
added
milestone/needs-approval
and removed
milestone/incomplete-labels
labels
|
Will it have been depreciated for 6 months as of the 1.11 release? I don't think it has been 6 months since the 1.10 release at this point... |
|
Reading it more closely, I guess it doesn't prevent us from changing defaults. @mtaufen WDYT? |
|
Exactly. We need to wait 6 months for removal (two releases, until v1.12), but to provide a smooth removal path we'll disable it already in v1.11. |
|
This seems reasonable to me. I feel like we should get an "ack" from a top-level API approver though. |
|
/retest |
1 similar comment
|
/retest |
k8s-ci-robot
added
the
lgtm
|
/test pull-kubernetes-e2e-kops-aws |
|
/test all [submit-queue is verifying that this PR is safe to merge] |
|
/test pull-kubernetes-e2e-kops-aws |
|
@luxas: The following test failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
Automatic merge from submit-queue (batch tested with PRs 63881, 64046, 63409, 63402, 63221). If you want to cherry-pick this change to another branch, please follow the instructions here. |
k8s-github-robot
pushed a commit
that referenced
this pull request
May 23, 2018
Automatic merge from submit-queue (batch tested with PRs 63914, 63887, 64116, 64026, 62933). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. kubeadm: Write kubelet config file to disk and persist in-cluster **What this PR does / why we need it**: In order to make configuration flow from the cluster level to node level, we need a way for kubeadm to tell the kubelet what config to use. As of v1.10 (I think) the kubelet can read `--config` using the kubelet Beta ComponentConfiguration API, so now we have an interface to talk to the kubelet properly. This PR: - Writes the kubelet ComponentConfig to `/var/lib/kubelet/config.yaml` on init and join - Writes an environment file to source in the kubelet systemd dropin `/var/lib/kubelet/kubeadm-flags.env`. This file contain runtime flags that should be passed to the kubelet. - Uploads a ConfigMap with the name `kubelet-config-1.X` - Patches the node object so that it starts using the ConfigMap with updates using Dynamic Kubelet Configuration, **only if the feature gate is set** (currently alpha and off by default, not intended to be switched on in v1.11) - Updates the phase commands to reflect this new flow The kubelet dropin file I used now looks like this: ``` # v1.11.x dropin as-is at HEAD # /etc/systemd/system/kubelet.service.d/10-kubeadm.conf --- [Service] Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf" Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml" EnvironmentFile-=/var/lib/kubelet/kubeadm-flags.env # Should default to 0 in v1.11: #63881, and hence not be here in the real v1.11 manifest Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0" # Should be configurable via the config file: #63878, and hence be configured using the file in v1.11 Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true" ExecStart= ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS --- # v1.11.x dropin end goal # /etc/systemd/system/kubelet.service.d/10-kubeadm.conf --- [Service] Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf" Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml" EnvironmentFile-=/var/lib/kubelet/kubeadm-flags.env ExecStart= ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS --- # Environment file dynamically created at runtime by "kubeadm init" # /var/lib/kubelet/kubeadm-flags.env KUBELET_KUBEADM_ARGS=--cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni ``` **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*: Fixes kubernetes/kubeadm#822 Fixes kubernetes/kubeadm#571 **Special notes for your reviewer**: **Release note**: ```release-note "kubeadm init" now writes a structured and versioned kubelet ComponentConfiguration file to `/var/lib/kubelet/config.yaml` and an environment file with runtime flags (you can source this file in the systemd kubelet dropin) to `/var/lib/kubelet/kubeadm-flags.env`. ``` @kubernetes/sig-cluster-lifecycle-pr-reviews @mtaufen
mvladev
pushed a commit
to mvladev/gardener
that referenced
this pull request
May 23, 2018
It should be removed in 1.12. kubernetes/kubernetes#63881
k8s-github-robot
pushed a commit
that referenced
this pull request
May 25, 2018
Automatic merge from submit-queue. Prevent 1.10 e2es testing deprecated CAdvisorPort in 1.11 **What this PR does / why we need it**: The public cadvisor port by default is disabled in #63881, targeted for v1.11. But 1.10 e2e tests get run against v1.11+ masters during upgrade tests. https://k8s-testgrid.appspot.com/sig-release-master-upgrade#gce-1.10-master-upgrade-cluster-parallel **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*: Fixes #64158 **Special notes for your reviewer**: /cc luxas BenTheElder krzyzacy **Release note**: ```release-note Prevent 1.10 e2es testing deprecated CAdvisorPort in v1.11 ```
k8s-github-robot
pushed a commit
that referenced
this pull request
Jul 7, 2018
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Remove --cadvisor-port - has been deprecated since v1.10 **What this PR does / why we need it**: **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*: Fixes #56523 **Special notes for your reviewer**: - Deprecated in #59827 (v1.10) - Disabled in #63881 (v1.11) **Release note**: ```release-note [action required] The formerly publicly-available cAdvisor web UI that the kubelet started using `--cadvisor-port` is now entirely removed in 1.12. The recommended way to run cAdvisor if you still need it, is via a DaemonSet. ```
|
It looks like this didn't make it into 1.11. Retagging as 1.12. |
|
@tallclair ?? This made it into v1.11 for sure. |
|
Huh, sorry. I don't know what I was looking at before. Maybe I got my tabs crossed. |
|
No problem |
|
@luxas do you have a PR against the 1.12 docs branch open for this? I have a feeling this will create a snowball of doc PRs, but maybe you could help find a spot for a general announcement? Thanks! |
cpick
pushed a commit
to cpick/prometheus-operator
that referenced
this pull request
Jan 31, 2019
K8s 1.11 [disables](kubernetes/kubernetes#63881) the separate cAdvisor port (4194) on kubelets. In GKE >=1.11 this data must now be fetched from the same port/endpoint as the rest of the metrics (under a separate path). Add a `httpSeparateCadvisor` setting to support this.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Quoting @tallclair in #56523:
Disabling the publicly-available cAdvisor port is beneficial for security, as you might not want to expose the UI with lots of information about what your system is doing. We already did this for all kubeadm deployments in v1.7, and haven't recieved any issues for that. This should be okay to do at this stage, as this flag was deprecated in v1.10. Given we need to support this flag for one more release (v1.11), it makes perfect sense to instead switch it off in preparation for v1.12 when we can delete it (see the deprecation policy)
Which issue(s) this PR fixes (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close the issue(s) when PR gets merged):Part of #56523
Special notes for your reviewer:
I removed the e2e test that expects cAdvisor to be running, as we don't expect it to be anymore.
Release note:
cc @kubernetes/sig-cluster-lifecycle-pr-reviews @kubernetes/sig-auth-pr-reviews @kubernetes/sig-node-pr-reviews