-
Notifications
You must be signed in to change notification settings - Fork 7.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CNI] CNI binary for 1.19.6 crashes if sh
is not on the host
#48746
Comments
@Smeb any chance you can try out #48757?
can do it (replace hub, probably) |
Awesome turnaround! Can do tomorrow (GMT). Will get back to you when I've had a chance to try it. |
Runs into a permission issue when changing just the image and nothing else:
Here are some logs from running on the node (just to show the binary is there, running the patched version, and executable):
Which looking in the kubelet logs is a result of
Something we can fix, but indicates that it may cause issues for other folks. |
Something else I saw is that in the case where we get an error, the logging doesn't work from inside |
For reference, this also occurs on |
So this is SELinux blocking any |
We don't have any custom SELinux rules set, so it's the default (we're using I can try to take a look at the configuration today. One other option might be using |
Small follow up - we also escalated this with AWS who are investigating. Might be they have a solution for the AWS Bottlerocket case that works for everyone. I'm a little concerned that we don't have a solution (yet). Is this a behaviour we could gate behind a flag? Obviously the ideal would be to find something that we know works for all environments, but I'm a little concerned that might be tricky given the diversity of environments out there. I don't like the idea of complicating the configuration, but currently Bottlerocket users can't easily use the latest supported patch releases of the currently supported versions as is. |
Regarding this error:
This is saying that However, if you did try to mount over
From a security policy standpoint, it doesn't make sense to allow a process from an unprivileged container (the If the mount attempt were treated as optional, that would work for Bottlerocket, which has a very generic |
Ignoring selinux, is this true given we are unsharing the mount namespace? With this, the mount shouldn't impact other processes? Even if thats true, selinux still may not take that into consideration, though.
This is my top preference but I couldn't find a great solution (LD_PRELOAD didn't seem great). Here is my current options in preference order:
I'll explore these a bit, but input from others welcome as well. I am trying to setup a bottlerocket env so I can reliably test each of these as well (if someone knows a way to run bottlerocket in docker like In the meantime, I would recommend users on bottlerocket do not upgrade to 1.19.6/1.20.2. While I cannot make promises, I am confident we will implement at least option 3 by the next patch release (so you shouldn't need to worry about being stuck on an old version and some security patch comes out that is incompatible, for instance). |
I got a bottlerocket setup and can reproduce
Parsing nsswitch.conf seems like the type of thing that might be error prone, so we could change "Come up with a way to detect if we need the mount, and only use it if so" to "Try to overwrite it, but do not exit on error". The downside is we will get these audit messages though |
One big question I had was why can we not I believe here we restrict CNI plugins: https://github.com/bottlerocket-os/bottlerocket/blob/7452c37e0b1abf19797a23bb7ab4d70bb0aef1cf/packages/selinux-policy/rules.cil#L90-L93 while the container runtime runs as another group (runtime_t) which has more access (maybe a bit off, first time dealing with SELinux rules...). So the CNI plugins essentially have less privilege than the container runtime? |
Yes, the container runtime is relatively privileged - it needs to be able to change the label of processes it launches, which is a bit like The answer to your first question is that SELinux is not namespace-aware, so it doesn't have a way to take the namespace into account when answering the question "does subject have permission to perform action on target ?" For this case, we might want to say "yes, as long as that action doesn't happen in the initial mount namespace" but there's no way to express that constraint.
Go does this for "hosts" to determine which DNS resolver it should use; if it finds a mechanism it doesn't understand, it switches to the cgo resolver so that the system library is used instead. In practice I expect you'll find that most Linux distros include an entry more complex than Bottlerocket's
I wouldn't worry about the audit message very much if it's just one AVC denial when the pod starts up. That is pretty common in my experience, and relatively harmless if the software has a fallback path. |
New iteration pushed up to #48757. This seems to work on GKE and Bottlerocket (where I have tested so far). Basically if we cannot setup the mounts we just continue anyways |
Thanks for all the follow up work. I just ran the latest version in our environment and can confirm it works. We get a lot of audit messages (as expected), but otherwise everything looks good. |
* cni: allow running in environments without sh/mount Fixes #48746 * wip * working * wip * Finally looking reasonabler * add release-note
* cni: allow running in environments without sh/mount Fixes istio#48746 * wip * working * wip * Finally looking reasonabler * add release-note (cherry picked from commit 0b32950)
* cni: allow running in environments without sh/mount Fixes istio#48746 * wip * working * wip * Finally looking reasonabler * add release-note (cherry picked from commit 0b32950) (cherry picked from commit 4a81bd6)
One warning to folks - between the regression in 1.18 and the fix, 1.18 is now EOL https://istio.io/latest/news/support/announcing-1.18-eol-final/. As such, this fix (and any other bugs or CVEs in Istio) will only be applied to 1.19+. 1.19 and 1.20 will be fixed in the next patch release (couple of weeks) |
* iptables: avoid `nsenter`ing twice (#48423) We are already in the network namespace as this point, no need to enter again (see `plugin.Program`) (cherry picked from commit e93c47d) (cherry picked from commit 7e8a5bd) * cni: allow running in environments without sh/mount (#48757) * cni: allow running in environments without sh/mount Fixes #48746 * wip * working * wip * Finally looking reasonabler * add release-note (cherry picked from commit 0b32950) (cherry picked from commit 4a81bd6)
* iptables: avoid `nsenter`ing twice (#48423) We are already in the network namespace as this point, no need to enter again (see `plugin.Program`) (cherry picked from commit e93c47d) * cni: allow running in environments without sh/mount (#48757) * cni: allow running in environments without sh/mount Fixes #48746 * wip * working * wip * Finally looking reasonabler * add release-note (cherry picked from commit 0b32950)
* cni: allow running in environments without sh/mount Fixes istio#48746 * wip * working * wip * Finally looking reasonabler * add release-note
* cni: allow running in environments without sh/mount Fixes istio#48746 * wip * working * wip * Finally looking reasonabler * add release-note
Hi, can you link in which patch release versions does this fix exist?
|
@howardjohn Is there a release with the fix? I see there are quite a few releases in the last weeks. |
fixed in 1.21.3 |
Is this the right place to submit this?
Bug Description
We're running into an issue when we try to use Istio 1.19.6 on our EKS clusters. 1.19.5 works. We think it's because of this patch #48422. We're running on bottlerocket, which doesn't include a shell - that change means that
exec.Command("sh" ...)
is called on both code paths, while previously it was only run for older versions of iptables. This blocks us from upgrading to1.19.6
(and presumably1.20.2
which includes that logic).The error we get (kubelet logs):
Not sure on the best fix if this is the issue. One (slightly hacky) option would be for the CNI binary to call itself to set up the mini-container environment, since that way we know the binary exists and can run.
Version
Additional Information
No response
The text was updated successfully, but these errors were encountered: