kubeadm: do not use --admission-control for the API server #64165

neolit123 · 2018-05-22T19:50:55Z

What this PR does / why we need it:
The API server argument --admission-control is deprecated.
Use the following arguments instead:
--enable-admission-plugins=NodeRestriction
--disable-admission-plugins=PersistentVolumeLabel

Add comment that PersistentVolumeLabel should be removed at some
point in 1.11.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Updates kubernetes/kubeadm#840

Special notes for your reviewer:
NONE

Release note:

kubeadm: when starting the API server use the arguments --enable-admission-plugins and --disable-admission-plugins instead of the deprecated --admission-control.

@luxas
@kubernetes/sig-cluster-lifecycle-pr-reviews
/area kubeadm

timothysc · 2018-05-22T21:02:15Z

cmd/kubeadm/app/phases/controlplane/manifests_test.go

@@ -154,7 +154,8 @@ func TestGetAPIServerCommand(t *testing.T) {
 			expected: []string{
 				"kube-apiserver",
 				"--insecure-port=0",
-				"--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota",
+				"--enable-admission-plugin=NodeRestriction",
+				"--disable-admission-plugin=PersistentVolumeLabel",


If this is deprecated I don't know why we need to explicitly specify.

@luxas can comment further i guess, i was just following the guide here:
kubernetes/kubeadm#840

It's still enabled by default which is a bug. Until that bug is fixed, let's leave it out here.

I'm fine with defaulting off, but lets cross-link an issue.

should i cross link to this (i.e. adding a comment with link in the code)?:
kubernetes/kubeadm#840

@neolit123 I'm thinking about the api-server deprecation default issue, should be in the main repo somewhere.

i can't seem to find it. only found the PR that deprecated admission-control: #58123

#64326 and #52617 and #52618

luxas

Thanks @neolit123! Can you use kubeadm with this patch to start up both a v1.10 cluster and a v1.11-beta.0 API server and post the final admission chain that was loaded (it's printed in the API server logs in the beginning), so we have a double-check that everything we want to be running actually is?
/approve

luxas · 2018-05-22T21:01:56Z

cmd/kubeadm/app/phases/controlplane/manifests.go

@@ -143,7 +143,8 @@ func getAPIServerCommand(cfg *kubeadmapi.MasterConfiguration) []string {
 	defaultArguments := map[string]string{
 		"advertise-address":               cfg.API.AdvertiseAddress,
 		"insecure-port":                   "0",
-		"admission-control":               defaultAdmissionControl,


please remove the unused variable

will do.

edit: done

neolit123 · 2018-05-22T21:31:45Z

Thanks @neolit123! Can you use kubeadm with this patch to start up both a v1.10 cluster and a v1.11-beta.0 API server and post the final admission chain that was loaded (it's printed in the API server logs in the beginning), so we have a double-check that everything we want to be running actually is?
/approve

ha!! as long as my control plane even starts. my setup is beyond broken ATM and it's not really kubeadm's fault. i can try...

stealthybox · 2018-05-22T22:08:34Z

@neolit123 if your personal lab is non-functional, this should serve the need:
https://github.com/kubernetes/kubeadm/tree/master/vagrant

just change the package versions in the Vagrantfile to 1.10.2 instead of 1.9.7 and then follow the guide for comparing different builds with the already installed kubeadm binary

neolit123 · 2018-05-22T22:13:38Z

yeah @stealthybox . we spoke about it at KubeCon.
i didn't had the time to try it, but i definitely need to find the time...

neolit123 · 2018-05-25T14:51:42Z

cmd/kubeadm/app/phases/controlplane/manifests.go

@@ -143,7 +141,9 @@ func getAPIServerCommand(cfg *kubeadmapi.MasterConfiguration) []string {
 	defaultArguments := map[string]string{
 		"advertise-address":               cfg.API.AdvertiseAddress,
 		"insecure-port":                   "0",
-		"admission-control":               defaultAdmissionControl,
+		// https://github.com/kubernetes/kubeadm/issues/840


^ added link to the issue here.

neolit123 · 2018-05-25T19:58:20Z

/test pull-kubernetes-kubemark-e2e-gce-big
/test pull-kubernetes-node-e2e

neolit123 · 2018-05-25T20:29:28Z

/test pull-kubernetes-node-e2e
/test pull-kubernetes-kubemark-e2e-gce-big

luxas

Please fix these comments. After that this LGTM

luxas · 2018-05-26T21:00:58Z

cmd/kubeadm/app/phases/controlplane/manifests.go

-		"admission-control":               defaultAdmissionControl,
+		"advertise-address": cfg.API.AdvertiseAddress,
+		"insecure-port":     "0",
+		// https://github.com/kubernetes/kubernetes/pull/58123


I'm not sure we should have this here...

luxas · 2018-05-26T21:01:33Z

cmd/kubeadm/app/phases/controlplane/manifests.go

+		"insecure-port":     "0",
+		// https://github.com/kubernetes/kubernetes/pull/58123
+		"enable-admission-plugin":         "NodeRestriction",
+		"disable-admission-plugin":        "PersistentVolumeLabel",


Add a TODO to remove this in kubeadm v1.11, as it's automatically disabled in v1.11, and reference #64326.
We can't skip it now as we support v1.10 clusters still.

luxas · 2018-05-26T21:02:28Z

cmd/kubeadm/app/phases/controlplane/manifests.go

+		"advertise-address": cfg.API.AdvertiseAddress,
+		"insecure-port":     "0",
+		// https://github.com/kubernetes/kubernetes/pull/58123
+		"enable-admission-plugin":         "NodeRestriction",


These are actually called enable/disable-admission-plugins

luxas · 2018-05-26T21:05:02Z

cmd/kubeadm/app/phases/controlplane/manifests_test.go

@@ -154,7 +154,8 @@ func TestGetAPIServerCommand(t *testing.T) {
 			expected: []string{
 				"kube-apiserver",
 				"--insecure-port=0",
-				"--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota",
+				"--enable-admission-plugin=NodeRestriction",
+				"--disable-admission-plugin=PersistentVolumeLabel",


#64326 and #52617 and #52618

neolit123 · 2018-05-26T22:29:23Z

updated.

fixed typo - missing s in argument keys
add TODO comment above PersistentVolumeLabel usage

added the same TODO note here and also made this PR not close the issue:
kubernetes/kubeadm#840

The API server argument --admission-control is deprecated. Use the following arguments instead: --enable-admission-plugins=NodeRestriction --disable-admission-plugins=PersistentVolumeLabel Add comment that PersistentVolumeLabel should be removed at some point in 1.11.

luxas

/lgtm

luxas · 2018-05-28T18:57:58Z

/retest

k8s-ci-robot · 2018-05-28T18:58:29Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: luxas, neolit123

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~cmd/kubeadm/OWNERS~~ [luxas]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

neolit123 · 2018-05-28T19:39:16Z

/test pull-kubernetes-e2e-gce-100-performance
/test pull-kubernetes-e2e-gce
/test pull-kubernetes-e2e-gce-device-plugin-gpu

fejta-bot · 2018-05-28T22:23:03Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

k8s-github-robot · 2018-05-29T01:20:09Z

Automatic merge from submit-queue (batch tested with PRs 64308, 64367, 64165, 64274). If you want to cherry-pick this change to another branch, please follow the instructions here.

Arnavion · 2018-06-27T23:43:15Z

Using minikube to create a 1.11.0 cluster puts this in /etc/kubernetes/manifests/kube-apiserver.yaml :

spec:
  containers:
  - command:
    - kube-apiserver
    - --admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota
    <snip>
    - --disable-admission-plugins=PersistentVolumeLabel
    - --enable-admission-plugins=NodeRestriction

so apiserver fails to start because admission-control and enable-admission-plugins/disable-admission-plugins flags are mutually exclusive

$ minikube version

minikube version: v0.28.0

and inside the cluster VM:

$ kubeadm version 

kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:14:41Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}

The enable/disable parameters are presumably coming because of this PR, so is there some place left that's still creating the default admission control parameter list?

neolit123 · 2018-06-27T23:47:30Z

the --admission-control part should not be there. it's unclear to me what's adding it.

Arnavion · 2018-06-28T00:12:56Z

Ah, it's coming from the MasterConfiguration/apiServerExtraArgs in the config file passed to kubeadm. So this is probably minikube creating the file incorrectly. Will follow up there.

neolit123 · 2018-06-28T00:13:35Z

@Arnavion
yes, these come from Minikube:
https://github.com/kubernetes/minikube/blob/master/pkg/util/constants.go#L35
https://github.com/kubernetes/minikube/blob/master/pkg/minikube/bootstrapper/kubeadm/versions.go#L205

this usage is out of date with k8s 1.11.

Arnavion · 2018-06-28T00:15:40Z

Yeah, just found it. I'll file a bug there.

k8s-ci-robot requested review from luxas and timothysc May 22, 2018 19:51

timothysc reviewed May 22, 2018

View reviewed changes

luxas approved these changes May 22, 2018

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 22, 2018

luxas self-assigned this May 22, 2018

neolit123 force-pushed the admission-control branch from 3cad80a to c16ff56 Compare May 22, 2018 22:08

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 25, 2018

neolit123 force-pushed the admission-control branch from c16ff56 to 876547f Compare May 25, 2018 14:43

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 25, 2018

neolit123 commented May 25, 2018

View reviewed changes

neolit123 force-pushed the admission-control branch from 876547f to 077ba44 Compare May 25, 2018 19:32

luxas reviewed May 26, 2018

View reviewed changes

luxas mentioned this pull request May 26, 2018

kubeadm should not use --admission-controller anymore. kubernetes/kubeadm#840

Closed

neolit123 force-pushed the admission-control branch from 077ba44 to c61004c Compare May 26, 2018 22:26

neolit123 force-pushed the admission-control branch from c61004c to 8d84ef6 Compare May 26, 2018 22:36

luxas approved these changes May 28, 2018

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 28, 2018

k8s-github-robot merged commit c56e049 into kubernetes:master May 29, 2018

neolit123 mentioned this pull request May 29, 2018

Remove usage of PersistentVolumeLabel in phases/controlplane/manifests*.go kubernetes/kubeadm#866

Closed

Arnavion mentioned this pull request Jun 28, 2018

Cannot deploy v1.11.0 because minikube puts bad default parameters in kube-apiserver manifest kubernetes/minikube#2942

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kubeadm: do not use --admission-control for the API server #64165

kubeadm: do not use --admission-control for the API server #64165

neolit123 commented May 22, 2018 •

edited

Loading

timothysc May 22, 2018

neolit123 May 22, 2018

luxas May 23, 2018

timothysc May 24, 2018

neolit123 May 24, 2018 •

edited

Loading

timothysc May 25, 2018

neolit123 May 25, 2018

luxas May 26, 2018

luxas left a comment

luxas May 22, 2018

neolit123 May 22, 2018 •

edited

Loading

neolit123 commented May 22, 2018

stealthybox commented May 22, 2018 •

edited

Loading

neolit123 commented May 22, 2018

neolit123 May 25, 2018

neolit123 commented May 25, 2018

neolit123 commented May 25, 2018

luxas left a comment

luxas May 26, 2018

luxas May 26, 2018

luxas May 26, 2018

luxas May 26, 2018

neolit123 commented May 26, 2018 •

edited

Loading

luxas left a comment

luxas commented May 28, 2018

k8s-ci-robot commented May 28, 2018

neolit123 commented May 28, 2018

fejta-bot commented May 28, 2018

k8s-github-robot commented May 29, 2018

Arnavion commented Jun 27, 2018

neolit123 commented Jun 27, 2018

Arnavion commented Jun 28, 2018 •

edited

Loading

neolit123 commented Jun 28, 2018

Arnavion commented Jun 28, 2018

kubeadm: do not use --admission-control for the API server #64165

kubeadm: do not use --admission-control for the API server #64165

Conversation

neolit123 commented May 22, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neolit123 May 24, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luxas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neolit123 May 22, 2018 • edited Loading

Choose a reason for hiding this comment

neolit123 commented May 22, 2018

stealthybox commented May 22, 2018 • edited Loading

neolit123 commented May 22, 2018

Choose a reason for hiding this comment

neolit123 commented May 25, 2018

neolit123 commented May 25, 2018

luxas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neolit123 commented May 26, 2018 • edited Loading

luxas left a comment

Choose a reason for hiding this comment

luxas commented May 28, 2018

k8s-ci-robot commented May 28, 2018

neolit123 commented May 28, 2018

fejta-bot commented May 28, 2018

k8s-github-robot commented May 29, 2018

Arnavion commented Jun 27, 2018

neolit123 commented Jun 27, 2018

Arnavion commented Jun 28, 2018 • edited Loading

neolit123 commented Jun 28, 2018

Arnavion commented Jun 28, 2018

neolit123 commented May 22, 2018 •

edited

Loading

neolit123 May 24, 2018 •

edited

Loading

neolit123 May 22, 2018 •

edited

Loading

stealthybox commented May 22, 2018 •

edited

Loading

neolit123 commented May 26, 2018 •

edited

Loading

Arnavion commented Jun 28, 2018 •

edited

Loading