make the kubelet cobra command complete #58405

deads2k · 2018-01-17T17:39:52Z

This pull attempts a move from the cmd/kubelet to the cobra command where it can re-used.

/assign @mtaufen
/assign @liggitt
@ncdc fyi

NONE

ncdc · 2018-01-17T17:42:54Z

cmd/kubelet/kubelet.go

-	}
-
-	// initialize logging and defer flush
+	utilflag.InitFlags()


Don't think you can do this - see

kubernetes/cmd/kube-proxy/proxy.go

Lines 36 to 38 in 8a6bb3e

// TODO: once we switch everything over to Cobra commands, we can go back to calling

// utilflag.InitFlags() (by removing its pflag.Parse() call). For now, we have to set the

// normalize func and add the go flag set by hand.

It seems to work ok for the scheduler: https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-scheduler/scheduler.go#L32-L34

It seems to work ok for the scheduler: https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-scheduler/scheduler.go#L32-L34

I see the bug. Registering global flags is naughty. Fixed.

As we discussed on slack, the scheduler is registering its flags globally into pflag.CommandLine. The kube-proxy registers its flags into the cobra command's .Flags(), which is why utilflag.InitFlags() doesn't work for kube-proxy but does for the scheduler. We need to fix the scheduler, finish converting all the executables to use cobra (#34732), then we can use InitFlags.

ncdc · 2018-01-17T17:43:18Z

cmd/kubelet/app/server.go

@@ -97,8 +98,9 @@ const (
 // NewKubeletCommand creates a *cobra.Command object with default parameters
 func NewKubeletCommand() *cobra.Command {
 	// ignore the error, as this is just for generating docs and the like


This comment isn't true any more

This comment isn't true any more

That is a code error. Updated to panic

ncdc · 2018-01-17T17:44:12Z

cmd/kubelet/app/server.go

+			// TODO(mtaufen): won't need this this once dynamic config is GA
+			// set feature gates so we can check if dynamic config is enabled
+			if err := utilfeature.DefaultFeatureGate.SetFromMap(kubeletServer.KubeletConfiguration.FeatureGates); err != nil {
+				glog.Fatal(err)


I know this was a move, but consider cmdutil.CheckErr() here and below?

I know this was a move, but consider cmdutil.CheckErr() here and below?

That will tie us to pkg/kubectl. I'd rather not end up with that linkage.

Ok, maybe we should consider taking that out of e.g. kube-proxy and (if it uses it) kube-scheduler in follow-ups.

Probably the base checkErr stuff should move out of kubectl.

liggitt · 2018-01-17T18:38:05Z

cmd/kubelet/kubelet.go

-	// set the normalize func, similar to k8s.io/apiserver/pkg/util/flag/flags.go:InitFlags
-	fs.SetNormalizeFunc(flag.WordSepNormalizeFunc)
-	// explicitly add flags from libs that register global flags
-	options.AddGlobalFlags(fs)


does the switch to cobra cmd expose us to picking up all the globally registered flags again?

does the switch to cobra cmd expose us to picking up all the globally registered flags again?

Nothing new that I know. It makes some of the previously registered ones more obvious via help since the help is accurate now, but it doesn't mess with the parsing.

this was selectively adding specific global flags, not all global flags, and marked most deprecated. does cobra just pick all the global ones up again?

this was selectively adding specific global flags, not all global flags, and marked most deprecated. does cobra just pick all the global ones up again?

It is consistent with the other commands. I'd much prefer to build a common whitelist if its a thing we want. After this, we have #58408 and then we can start dealing with all the command uniformally and fix it everywhere.

I don't think we should take a step backwards just to be "consistent" with other commands - which are probably doing it wrong by leaking flags anyway.

Ok, I can update this tomorrow morning

mtaufen

I think you mostly got things in the right place, but we should try to accomplish this without regressing on the Kubelet's existing behavior.

mtaufen · 2018-01-17T22:50:26Z

cmd/kubelet/kubelet.go

-	// set the normalize func, similar to k8s.io/apiserver/pkg/util/flag/flags.go:InitFlags
-	fs.SetNormalizeFunc(flag.WordSepNormalizeFunc)
-	// explicitly add flags from libs that register global flags
-	options.AddGlobalFlags(fs)


I don't think we should take a step backwards just to be "consistent" with other commands - which are probably doing it wrong by leaking flags anyway.

mtaufen · 2018-01-17T22:51:22Z

cmd/kubelet/kubelet.go

+	// utilflag.InitFlags() (by removing its pflag.Parse() call). For now, we have to set the
+	// normalize func and add the go flag set by hand.
+	pflag.CommandLine.SetNormalizeFunc(utilflag.WordSepNormalizeFunc)
+	pflag.CommandLine.AddGoFlagSet(goflag.CommandLine)


Please don't leak the global command line into the Kubelet here. We already did a lot of work to fix this for the Kubelet in #57613. This will be a regression.

Why we can't just construct the private flagset in Run (above) and apply the passed in args []string to it, just like we do in main today. Effectively, why can't you just copy the body of today's main to Run, and substitute Run's args for today's os.Args? Does cobra do something special that gets in the way?

That's non-cobra-y and doesn't reflect the patterns we've found to work well in kubectl and openshift. I think you'll be happier take a different tack that separates your data and transformation phases more cleanly. This doesn't preclude such a follow-up or make it harder to do, but you should carefully consider the data flow.

mtaufen · 2018-01-17T23:06:47Z

cmd/kubelet/app/server.go

@@ -121,8 +124,52 @@ is checked every 20 seconds (also configurable with a flag).
 HTTP server: The kubelet can also listen for HTTP and respond to a simple API
 (underspec'd currently) to submit a new manifest.`,
 		Run: func(cmd *cobra.Command, args []string) {


We will need to make sure it's still possible for the Kubelet to re-parse its command line multiple times during its bootstrap, see #56995 and #56171. I'd suppose this can be done with the args []string passed in here but need to make sure.

Reparsing is possible. I'll link you to some OpenShift code that does it. In general, be careful about doing it.

mtaufen · 2018-01-17T23:11:46Z

cmd/kubelet/app/server.go

+			}
+
+			// construct a KubeletServer from kubeletFlags and kubeletConfig
+			kubeletServer.KubeletConfiguration = *kubeletConfig


I'd prefer we construct the kubeletFlags and kubeletConfig independently, and construct the kubeletServer just before we pass it to Run. The code reads more cleanly if it consists of a sequence of constructions rather than a sequence of mutations to a pre-constructed object.

I'd like that too, but I think the order should be 1) split the flag struct out, 2) write code to go from structul to runtime, 3) update command

mtaufen · 2018-01-17T23:12:30Z

cmd/kubelet/app/server.go

-	// ignore the error, as this is just for generating docs and the like
-	s, _ := options.NewKubeletServer()
-	s.AddFlags(pflag.CommandLine)
+	kubeletServer, err := options.NewKubeletServer()


Why can't we construct this inside Run, just like it's constructed in main() today?

You need to use it to later add flags.

mtaufen · 2018-01-17T23:46:19Z

/approve
/hold
(hold for global flags changes, as discussed on slack)
ping me on slack when it's ready, thanks for understanding

derekwaynecarr · 2018-01-17T23:52:14Z

/approve
/hold

deads2k · 2018-01-18T13:27:48Z

Global flags suppressed. Made it slightly more forward building too.

deads2k · 2018-01-18T13:28:38Z

/approve no-issue

deads2k · 2018-01-18T13:33:43Z

/approve
/hold
(hold for global flags changes, as discussed on slack)
ping me on slack when it's ready, thanks for understanding

global flags update and managed to move the kubeletServer instantiation down, though now there are two sources for flags (kubeletconfiguration and kubeletflags)

/hold cancel

mfojtik · 2018-01-18T14:57:03Z

cmd/kubelet/kubelet.go

-	if err := app.Run(kubeletServer, kubeletDeps); err != nil {
-		die(err)
+	if err := command.Execute(); err != nil {
+		os.Exit(1)


where I can see the error?

mfojtik · 2018-01-18T14:59:40Z

/lgtm

deads2k · 2018-01-18T16:01:45Z

/retest

mtaufen · 2018-01-18T17:46:37Z

/lgtm

k8s-ci-robot · 2018-01-18T17:46:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, derekwaynecarr, mfojtik, mtaufen

Associated issue: #34732

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

~~cmd/kubelet/OWNERS~~ [derekwaynecarr]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

mtaufen · 2018-01-18T17:47:52Z

not sure what's going on with that GPU test... maybe the nvidia driver install flaked on that one node?
/retest

k8s-github-robot · 2018-01-18T18:27:39Z

Automatic merge from submit-queue (batch tested with PRs 58209, 57561, 58405). If you want to cherry-pick this change to another branch, please follow the instructions here.

k8s-ci-robot assigned liggitt and mtaufen Jan 17, 2018

deads2k added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Jan 17, 2018

ncdc reviewed Jan 17, 2018

View reviewed changes

deads2k force-pushed the kubelet-01-start branch 2 times, most recently from 35c0111 to 126cdba Compare January 17, 2018 18:09

liggitt reviewed Jan 17, 2018

View reviewed changes

deads2k mentioned this pull request Jan 17, 2018

switch hyperkube to cobra #58408

Merged

mtaufen requested changes Jan 17, 2018

View reviewed changes

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 17, 2018

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 18, 2018

deads2k force-pushed the kubelet-01-start branch from 126cdba to ff8002a Compare January 18, 2018 13:26

k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 18, 2018

k8s-ci-robot added do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Jan 18, 2018

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 18, 2018

mfojtik reviewed Jan 18, 2018

View reviewed changes

k8s-ci-robot assigned mfojtik Jan 18, 2018

k8s-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jan 18, 2018

make the kubelet cobra command complete

d113f0f

deads2k added lgtm Indicates that a PR is ready to be merged. and removed lgtm Indicates that a PR is ready to be merged. labels Jan 18, 2018

deads2k force-pushed the kubelet-01-start branch from ff8002a to d113f0f Compare January 18, 2018 15:14

mtaufen approved these changes Jan 18, 2018

View reviewed changes

k8s-github-robot merged commit 7f6dae7 into kubernetes:master Jan 18, 2018

deads2k mentioned this pull request Jan 18, 2018

Use cobra #58472

Closed

deads2k deleted the kubelet-01-start branch July 3, 2018 18:02

	// TODO: once we switch everything over to Cobra commands, we can go back to calling
	// utilflag.InitFlags() (by removing its pflag.Parse() call). For now, we have to set the
	// normalize func and add the go flag set by hand.

make the kubelet cobra command complete #58405

make the kubelet cobra command complete #58405

Conversation

deads2k commented Jan 17, 2018 • edited

Choose a reason for hiding this comment

deads2k Jan 17, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtaufen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtaufen commented Jan 17, 2018

derekwaynecarr commented Jan 17, 2018

deads2k commented Jan 18, 2018

deads2k commented Jan 18, 2018

deads2k commented Jan 18, 2018

Choose a reason for hiding this comment

mfojtik commented Jan 18, 2018

deads2k commented Jan 18, 2018

mtaufen commented Jan 18, 2018

k8s-ci-robot commented Jan 18, 2018

mtaufen commented Jan 18, 2018

k8s-github-robot commented Jan 18, 2018

deads2k commented Jan 17, 2018 •

edited

deads2k Jan 17, 2018 •

edited