Fix init order during starup for dynamic kubelet config

[odinuge]

What type of PR is this?
/kind bug

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #83183

Special notes for your reviewer:
This is just a "proof of concept" fix in order to make it easier to understand the problem. From what I understand it does work, and all the tests pass, but I don't think the is the correct solution long term. If this makes sense and works, I guess
we can use this fix on the v1.16 branch to mitigate the problem.

The probelm is essentially that the sync function changed in the PR is executed before the kubelet is started and the metrics are "Created" with the way #81534 works.

If the metrics ain't created, the values will not be registered, and the
metrics will not be visible in the metric endpoint.

Feel free to change the priority; but since this breaks all the related tests, and the metrics are missing, I guess this is ok
/priority critical-urgent
/sig node

Does this PR introduce a user-facing change?:

Fix error where metrics related to dynamic kubelet config isn't registered

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[odinuge]

As you see here, the node e2e tests (running with FOCUS="DynamicKubeletConfig" SKIP="") pass (note that they take 2hours to run 😅).

Non of them pass in master: https://testgrid.k8s.io/sig-node-kubelet#node-kubelet-serial

/cc @alejandrox1

[liggitt]

/assign @logicalhan

[logicalhan]

If I understand this correctly, then the metrics do show up eventually, right? Once the kubelet is created, then then metrics should reflect the syncs.

[odinuge]

@logicalhan If I understand this correctly, then the metrics do show up eventually, right? Once the kubelet is created, then then metrics should reflect the syncs.

I think #83184 (comment) answers this question.
Also, the tests using the metrics in master still fails, so it looks like they never appear.

[logicalhan]

Why can't we just move https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/server.go#L593-L598 down to below where the kubelet is actually initialized?

[odinuge]

Thanks for looking at it @logicalhan!

Why can't we just move
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/server.go#L593-L598 down to > below where the kubelet is actually initialized?

Yes, I think that is a good solution. I just need confirmation from someone that it will work, and that there is no dependencies that require it to stay where it is. I have updated the PR now, and it looks like the e2e tests pass too (the first few passed, but the tests takes 2 hours to complete).. All DynamicKubeletConfig node-e2e tests pass with this PR.

Metrics are lazily instantiated now to allow us to disable them. We've had issues in the past with high cardinality metrics (#73587). In order to disable metrics, we need to no-opt creation of metrics until they actually become registered, which is the current behavior. We can force metrics initialization like so (logicalhan/kubernetes@master...kubelet-metrics-force-initialization), but I really don't want to do that.

Agree that force-registering the metrics is a bad solution, and goes against the new metric stuff; so I don't think that is a viable solution to the issue.

It looks to me that these tests are failing because they are literally testing that the metrics are immediately registered. That seems like a weird thing to test.

The test has a 1min timeout when looking for the metrics. And as I said in the other comments; from what I see and understand, if the first metric write is lost, the metrics will not be reapplied (Sync will not run) unless some of the data is changed.

[odinuge]

cc @kubernetes/sig-node-bugs

[odinuge]

Ping @logicalhan. Any new thoughts?

[odinuge]

/cc mtaufen derekwaynecarr

[logicalhan]

/lgtm

[odinuge]

/assign dchen1107 derekwaynecarr

[liggitt]

all synchronous setup is done in Controller#Bootstrap(), everything done by this method (other than programming error sanity checks that exit the process) is done in goroutines

/lgtm

[liggitt]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt, odinuge

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• cmd/kubelet/OWNERS [liggitt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[odinuge]

/test pull-kubernetes-integration