Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add default sysctls to allow privileged ports with no capabilities #18421

Merged
merged 3 commits into from Mar 18, 2024

Conversation

prezha
Copy link
Collaborator

@prezha prezha commented Mar 17, 2024

this pr adds net.ipv4.ip_unprivileged_port_start sysctls as default to containerd and cri-o container runtimes to allow privileged ports to be used by containers run by a non-root user for kubernetes v1.22+

background

during testing the bump of kubernetes default version to v1.29.2, we discovered that coredns would not come up if containerd or cri-o container runtimes are used with the docker driver (example: conainerd and cri-o)

the coredns errored with: Listen: listen tcp :53: bind: permission denied

kubernetes v1.29.0+ uses coredns v1.11.1 and from v1.11.0, coredns runs as non-root user

some additional details about the issue also observed elsewhere can be found eg, here and here

since starting from kubernetes v1.22, the required net.ipv4.ip_unprivileged_port_start sysctl was marked as safe, but it's up to the container runtime to implement it - docker already has it enabled by default, whereas other CRIs have that as option but it's not enabled by default (containerd and cri-o) and this pr tries to address those two and enable corresponding options

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Mar 17, 2024
@k8s-ci-robot k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Mar 17, 2024
@afbjorklund
Copy link
Collaborator

afbjorklund commented Mar 17, 2024

Seems like this is something that should at least be documented upstream, for changing the runtime settings...

https://kubernetes.io/docs/setup/production-environment/container-runtimes/

@afbjorklund
Copy link
Collaborator

@prezha
Copy link
Collaborator Author

prezha commented Mar 17, 2024

there's a notion to go even further - by k8s requiring this from cri: kubernetes/kubernetes#102612 (comment)

also, containerd plans to have it enabled by default in v2.0: containerd/containerd#6924

@afbjorklund
Copy link
Collaborator

I don't think that is a big issue, just wondering if minikube should follow rather than lead?

@prezha
Copy link
Collaborator Author

prezha commented Mar 18, 2024

/ok-to-test

@k8s-ci-robot k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Mar 18, 2024
@minikube-pr-bot

This comment has been minimized.

@minikube-pr-bot

This comment has been minimized.

Copy link
Member

@spowelljr spowelljr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just two super small things in the comments

pkg/minikube/cruntime/crio.go Outdated Show resolved Hide resolved
pkg/minikube/cruntime/crio.go Outdated Show resolved Hide resolved
prezha and others added 2 commits March 18, 2024 20:11
Co-authored-by: Steven Powell <44844360+spowelljr@users.noreply.github.com>
Co-authored-by: Steven Powell <44844360+spowelljr@users.noreply.github.com>
@prezha
Copy link
Collaborator Author

prezha commented Mar 18, 2024

eagle eye 🙂
thanks @spowelljr
done!

@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 18421) |
+----------------+----------+---------------------+
| minikube start | 52.9s    | 52.7s               |
| enable ingress | 26.1s    | 23.8s               |
+----------------+----------+---------------------+

Times for minikube start: 50.4s 52.2s 53.1s 56.0s 53.0s
Times for minikube (PR 18421) start: 53.1s 53.8s 54.1s 51.7s 50.8s

Times for minikube ingress: 25.7s 26.1s 25.6s 25.7s 27.6s
Times for minikube (PR 18421) ingress: 25.7s 23.1s 25.5s 22.2s 22.6s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 18421) |
+----------------+----------+---------------------+
| minikube start | 23.6s    | 24.0s               |
| enable ingress | 20.3s    | 20.2s               |
+----------------+----------+---------------------+

Times for minikube ingress: 20.3s 20.3s 20.3s 20.3s 20.3s
Times for minikube (PR 18421) ingress: 20.8s 20.3s 20.3s 19.3s 20.3s

Times for minikube start: 22.5s 22.6s 25.6s 25.1s 22.1s
Times for minikube (PR 18421) start: 22.9s 22.5s 26.4s 25.7s 22.6s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 18421) |
+----------------+----------+---------------------+
| minikube start | 21.6s    | 23.3s               |
| enable ingress | 33.8s    | 30.7s               |
+----------------+----------+---------------------+

Times for minikube start: 21.9s 21.1s 20.4s 20.8s 23.6s
Times for minikube (PR 18421) start: 21.4s 22.3s 24.5s 23.9s 24.3s

Times for minikube ingress: 30.8s 31.3s 31.3s 29.8s 45.8s
Times for minikube (PR 18421) ingress: 30.8s 30.8s 29.8s 30.8s 31.3s

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: prezha, spowelljr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@spowelljr spowelljr merged commit ccbaa19 into kubernetes:master Mar 18, 2024
24 of 38 checks passed
@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
Docker_Linux_crio TestMissingContainerUpgrade (gopogh) 0.00 (chart)
KVM_Linux_crio TestAddons/parallel/LocalPath (gopogh) 0.63 (chart)
QEMU_macOS TestAddons/Setup (gopogh) 25.00 (chart)
QEMU_macOS TestErrorSpam/setup (gopogh) 25.00 (chart)
QEMU_macOS TestFunctional/parallel/CpCmd (gopogh) 25.00 (chart)
QEMU_macOS TestFunctional/parallel/FileSync (gopogh) 25.00 (chart)
QEMU_macOS TestFunctional/parallel/NonActiveRuntimeDisabled (gopogh) 25.00 (chart)
QEMU_macOS TestFunctional/parallel/SSHCmd (gopogh) 25.00 (chart)
QEMU_macOS TestFunctional/parallel/UpdateContextCmd/no_clusters (gopogh) 25.00 (chart)
QEMU_macOS TestFunctional/parallel/UpdateContextCmd/no_minikube_cluster (gopogh) 25.00 (chart)
QEMU_macOS TestFunctional/parallel/Version/components (gopogh) 25.00 (chart)
QEMU_macOS TestFunctional/serial/LogsCmd (gopogh) 25.00 (chart)
QEMU_macOS TestFunctional/serial/LogsFileCmd (gopogh) 25.00 (chart)
QEMU_macOS TestFunctional/parallel/CertSync (gopogh) 27.78 (chart)
QEMU_macOS TestFunctional/parallel/DashboardCmd (gopogh) 27.78 (chart)
QEMU_macOS TestFunctional/parallel/DockerEnv/bash (gopogh) 27.78 (chart)
QEMU_macOS TestFunctional/parallel/ImageCommands/ImageBuild (gopogh) 27.78 (chart)
QEMU_macOS TestFunctional/parallel/ImageCommands/ImageListJson (gopogh) 27.78 (chart)
QEMU_macOS TestFunctional/parallel/ImageCommands/ImageListShort (gopogh) 27.78 (chart)
QEMU_macOS TestFunctional/parallel/ImageCommands/ImageListTable (gopogh) 27.78 (chart)
QEMU_macOS TestFunctional/parallel/ImageCommands/ImageListYaml (gopogh) 27.78 (chart)
QEMU_macOS TestFunctional/parallel/ImageCommands/ImageLoadDaemon (gopogh) 27.78 (chart)
QEMU_macOS TestFunctional/parallel/ImageCommands/ImageLoadFromFile (gopogh) 27.78 (chart)
QEMU_macOS TestFunctional/parallel/ImageCommands/ImageReloadDaemon (gopogh) 27.78 (chart)
QEMU_macOS TestFunctional/parallel/ImageCommands/ImageSaveToFile (gopogh) 27.78 (chart)
QEMU_macOS TestFunctional/parallel/NodeLabels (gopogh) 27.78 (chart)
QEMU_macOS TestFunctional/parallel/PersistentVolumeClaim (gopogh) 27.78 (chart)
QEMU_macOS TestFunctional/parallel/ServiceCmd/DeployApp (gopogh) 27.78 (chart)
QEMU_macOS TestFunctional/parallel/ServiceCmd/Format (gopogh) 27.78 (chart)
QEMU_macOS TestFunctional/parallel/ServiceCmd/HTTPS (gopogh) 27.78 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants