Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix serialization of EnforceNodeAllocatable #44606

Conversation

ivan4th
Copy link
Contributor

@ivan4th ivan4th commented Apr 18, 2017

EnforceNodeAllocatable being nil and [] are treated in different
ways by kubelet. Namely, nil is replaced with []string{"pods"} by
the defaulting mechanism.

E.g. if you run kubelet in Docker-in-Docker environment
you may need to run it with the following options:
--cgroups-per-qos=false --enforce-node-allocatable=
(this corresponds to EnforceNodeAllocatable being empty array and not
null) If you then grab kubelet configuration via /configz and try to
reuse it for dynamic kubelet config, kubelet will think that
EnforceNodeAllocatable is null, failing to run in the
Docker-in-Docker environment.

Encountered this while updating Virtlet for Kubernetes 1.6
(the dev environment is based on kubeadm-dind-cluster)

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Apr 18, 2017
@k8s-reviewable
Copy link

This change is Reviewable

@k8s-github-robot k8s-github-robot added kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. release-note-label-needed labels Apr 18, 2017
@ivan4th ivan4th force-pushed the fix-serialization-of-enforce-node-allocatable branch 2 times, most recently from 2a13305 to a13a31e Compare April 18, 2017 12:31
@k8s-github-robot k8s-github-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Apr 18, 2017
@k8s-github-robot k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 28, 2017
@yujuhong yujuhong assigned mtaufen and unassigned mwielgus Apr 28, 2017
@yujuhong yujuhong added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-label-needed labels Apr 28, 2017
@yujuhong
Copy link
Contributor

Change looks fine to me, but adding @mtaufen since he's the authority of kubelet config.

@mtaufen
Copy link
Contributor

mtaufen commented May 8, 2017

/lgtm

This looks good to me for now. But keep in mind you really shouldn't be using the existing dynamic config implementation for anything other than testing (it's still alpha). It will also change significantly in 1.7: kubernetes/enhancements#281

Sorry for lateness on this review.

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 8, 2017
EnforceNodeAllocatable being `nil` and `[]` are treated in different
ways by kubelet. Namely, `nil` is replaced with `[]string{"pods"}` by
the defaulting mechanism.

E.g. if you run kubelet in Docker-in-Docker environment
you may need to run it with the following options:
`--cgroups-per-qos=false --enforce-node-allocatable=`
(this corresponds to EnforceNodeAllocatable being empty array and not
null) If you then grab kubelet configuration via /configz and try to
reuse it for dynamic kubelet config, kubelet will think that
EnforceNodeAllocatable is null, failing to run in the
Docker-in-Docker environment.

Encountered this while updating Virtlet for Kubernetes 1.6
(the dev environment is based on kubeadm-dind-cluster)
@ivan4th ivan4th force-pushed the fix-serialization-of-enforce-node-allocatable branch from a13a31e to ef85747 Compare May 16, 2017 23:12
@k8s-github-robot k8s-github-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels May 16, 2017
@ivan4th
Copy link
Contributor Author

ivan4th commented May 16, 2017

@mtaufen thanks for the comment. I've rebased the PR. We use dynamic kubelet config during the deployment of our CRI implementation (Virtlet), but it'll require some updates anyway to be compatible with forthcoming k8s 1.7, and as of now it looks like we'll be able to use the new dynamic kubelet config mechanism, too.

@ivan4th
Copy link
Contributor Author

ivan4th commented May 17, 2017

@k8s-bot kubemark e2e test this

@ivan4th
Copy link
Contributor Author

ivan4th commented May 17, 2017

@k8s-bot gce etcd3 e2e test this

@mtaufen
Copy link
Contributor

mtaufen commented May 19, 2017

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 19, 2017
@k8s-github-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ivan4th, mtaufen

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@mtaufen
Copy link
Contributor

mtaufen commented May 19, 2017

@ivan4th The Kubelet's configuration API is still alpha-grade, and likely to undergo violent breaking changes. At present, there is no guarantee that any field you are dynamically updating will still be dynamic in any future version. For this reason, I caution against using dynamic config for anything other than tests (even there, please use it sparingly, as I'm the one who gets to update everyone else's code when the API changes).

@ivan4th
Copy link
Contributor Author

ivan4th commented May 19, 2017

@mtaufen thanks! Well, our idea is giving users a simple way to try out Virtlet. It's CRI implementation that itself runs as a DaemonSet and in case of "bootstrap" mode (not intended for production!) it uses dynamic kubelet config to set remoteRuntimeEndpoint and remoteImageEndpoint. This makes Virtlet installation a matter of kubectl create and doesn't tie it to particular systemd unit structure etc. Any chance these settings will be available in future? Not asking for any guarantees, of course.

@ivan4th
Copy link
Contributor Author

ivan4th commented May 19, 2017

(running CRI impl as a DaemonSet is possible thanks to the CRI proxy mechanism that makes it easy to have multiple runtimes on the node)

@k8s-github-robot
Copy link

Automatic merge from submit-queue (batch tested with PRs 44606, 46038)

@k8s-github-robot k8s-github-robot merged commit 69ce7d8 into kubernetes:master May 19, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants