Linkerd without CNI - run as non-root #5505

part-time-githubber · 2021-01-11T06:29:43Z

Feature Request

What problem are you trying to solve?

We are not able to run Linkerd with CNI as another CNI we have in the chain does not play well with Linkerds. Hence we want to switch to Linkerd without CNI. But that needs our kube PSPs to allow

capability NET_RAW
capability NET_ADMIN
run as root

We are OK with the first two capabilities confined to the linkerd proxy binary without needing to run as root. That would be acceptable to InformationSecurity team at the place we are trying to run Linkerd. Is it possible to do that?

How should the problem be solved?

linkerd proxy container section added by mutation webhook will need to include something like
securityContext:
capabilities:
add:
- NET_RAW
- NET_ADMIN

the proxy image will need to be built with something like
RUN apt-get install libcap2-bin=1:2.25-2 -y --no-install-recommends
&& setcap 'cap_net_raw=+ep' \
&& setcap 'cap_net_admin=+ep'

What do you want to happen? Add any considered drawbacks.

Any alternatives you've considered?

Is there another way to solve this problem that isn't as good a solution?

Run all application workloads as Root which opens a big security hole

How would users interact with this feature?

If you can, explain how users will be able to use this. Maybe some sample CLI
output?

Kubernetes PSP for application workloads which needs to allow above two capabilities and can still require binary to NOT run as root

adleong · 2021-01-11T18:59:32Z

Hey @pankajmt! If we could run the proxy-init container as non-root but still with the appropriate net capabilities, we'd be open to it. As far as I know, no one has tried this so I'm not totally sure if it is possible or not. If you are interested in working on this, I'd love to take a look.

part-time-githubber · 2021-01-11T23:09:06Z

ah yes, it would be the proxy-init container and not the proxy container to be precise. I might start looking into this today.

part-time-githubber · 2021-01-12T12:29:50Z

Not much luck with attempt 1

[23:26] [ptolani@Pankajs-MacBook-Pro:~/greaterbank/infosec/PE-1247/pankajmt]$ [._.] cd linkerd2
[23:26] [ptolani@Pankajs-MacBook-Pro:~/greaterbank/infosec/PE-1247/pankajmt/linkerd2]$ [._.] git diff
diff --git a/charts/partials/templates/_proxy-init.tpl b/charts/partials/templates/_proxy-init.tpl
index c818e713..e821f9dd 100644
--- a/charts/partials/templates/_proxy-init.tpl
+++ b/charts/partials/templates/_proxy-init.tpl
@@ -44,8 +44,8 @@ securityContext:
   privileged: false
   {{- end }}
   readOnlyRootFilesystem: true
-  runAsNonRoot: false
-  runAsUser: 0
+  runAsNonRoot: true
+  runAsUser: 70000
 terminationMessagePolicy: FallbackToLogsOnError
 {{- if or (not .Values.global.cniEnabled) .Values.global.proxyInit.saMountPath }}
 volumeMounts:
@@ -58,5 +58,5 @@ volumeMounts:
 - mountPath: {{.Values.global.proxyInit.saMountPath.mountPath}}
   name: {{.Values.global.proxyInit.saMountPath.name}}
   readOnly: {{.Values.global.proxyInit.saMountPath.readOnly}}
-{{- end -}}  
+{{- end -}}
 {{- end -}}

[23:28] [ptolani@Pankajs-MacBook-Pro:~/greaterbank/infosec/PE-1247/pankajmt]$ [._.] cd linkerd2-proxy-init/
[23:28] [ptolani@Pankajs-MacBook-Pro:~/greaterbank/infosec/PE-1247/pankajmt/linkerd2-proxy-init]$ [._.] git diff
diff --git a/Dockerfile b/Dockerfile
index 154d9bc..6b1e2a0 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -18,9 +18,14 @@ RUN apt-get update \
     && apt-get install -y --no-install-recommends \
         iptables \
         procps \
+        libcap2-bin \
     && rm -rf /var/lib/apt/lists/* \
     && update-alternatives --set iptables /usr/sbin/iptables-legacy \
     && update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
+
+RUN groupadd -r linkerd && useradd -r -g linkerd linkerd
 COPY LICENSE /linkerd/LICENSE
 COPY --from=golang /out/linkerd2-proxy-init /usr/local/bin/proxy-init
+RUN setcap 'cap_net_raw,cap_net_admin=+ep' /usr/local/bin/proxy-init
+USER linkerd
 ENTRYPOINT ["/usr/local/bin/proxy-init"]

See this in startup of linkerd control plane

│ 2021/01/12 05:59:33 :; iptables -t nat -N PROXY_INIT_REDIRECT -m comment --comment proxy-init/redirect-common-chain/16104311 │
│ 73                                                                                                                           │
│ 2021/01/12 05:59:33 iptables v1.8.2 (legacy): can't initialize iptables table `nat': Permission denied (you must be root)    │
│ Perhaps iptables or your kernel needs to be upgraded.                                                                        │
│                                                                                                                              │
│ 2021/01/12 05:59:33 Aborting firewall configuration                                                                          │
│ Error: exit status 3                                                                                                         │
│ Usage:                                                                                                                       │
│   proxy-init [flags]                                                                                                         │
│                                                                                                                              │
│ Flags:                                                                                                                       │
│   -h, --help                               help for proxy-init                                                               │
│       --inbound-ports-to-ignore strings    Inbound ports and/or port ranges (inclusive) to ignore and not redirect to proxy. │
│  This has higher precedence than any other parameters.                                                                       │
│   -p, --incoming-proxy-port int            Port to redirect incoming traffic (default -1)                                    │
│       --netns string                       Optional network namespace in which to run the iptables commands                  │
│       --outbound-ports-to-ignore strings   Outbound ports and/or port ranges (inclusive) to ignore and not redirect to proxy │
│ . This has higher precedence than any other parameters.                                                                      │

part-time-githubber · 2021-01-13T05:23:30Z

Not much luck with attempt 2

Updated Dockerfile to allow child processes inherit the capabilities too and also assign capabilities also to iptables binary. Same output. Wonder if iptables 1.8.2 legacy on debian buster is good enough. i see mentions buster also now has nftables, successor to iptables.

[16:23] [ptolani@Pankajs-MacBook-Pro:~/greaterbank/infosec/PE-1247/pankajmt/linkerd2-proxy-init]$ [._.] git diff Dockerfile
diff --git a/Dockerfile b/Dockerfile
index 154d9bc..8a0d098 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -18,9 +18,16 @@ RUN apt-get update \
     && apt-get install -y --no-install-recommends \
         iptables \
         procps \
+        libcap2-bin \
     && rm -rf /var/lib/apt/lists/* \
     && update-alternatives --set iptables /usr/sbin/iptables-legacy \
     && update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
+
+RUN groupadd -r linkerd && useradd -r -g linkerd linkerd
 COPY LICENSE /linkerd/LICENSE
 COPY --from=golang /out/linkerd2-proxy-init /usr/local/bin/proxy-init
+RUN setcap cap_net_raw,cap_net_admin+eip /usr/local/bin/proxy-init
+RUN setcap cap_net_raw,cap_net_admin+eip /usr/sbin/xtables-legacy-multi
+USER linkerd
 ENTRYPOINT ["/usr/local/bin/proxy-init"]

part-time-githubber · 2021-01-13T12:57:11Z

Looks like I am able to run iptables commands in a pod without being root if the associated PSP in addition to capabilities NET_ADMIN and NET_RAW also has allowPrivilegeEscalation: true

Now as per https://kubernetes.io/docs/concepts/policy/pod-security-policy/,

This bool directly controls whether the no_new_privs flag gets set on the container process. This flag will prevent setuid binaries from changing the effective user ID, and prevent files from enabling extra capabilities (e.g. it will prevent the use of the ping tool). This behavior is required to effectively enforce MustRunAsNonRoot.

So, my understanding is, although not root, with your own binary, you can do things like root by including this in a docker image with setuid bit set or additional capabilities. so equally risky 👎

wmorgan · 2021-01-15T15:07:47Z

@pankajmt Thanks for the experimentation. To confirm, the only way you've found to have the proxy-init container to run as non-root is by setting allowPrivilegeEscalation to true in the PSP?

part-time-githubber · 2021-01-16T01:58:38Z

Following linkerd control plane PSP changes allow iptables to run as NON root

<   allowPrivilegeEscalation: false
---
>   allowPrivilegeEscalation: true
29c19
<     rule: RunAsAny
---
>     rule: MustRunAsNonRoot

my docker image where I have this working atm is

FROM ubuntu

RUN apt-get update && apt-get install -y --no-install-recommends iptables libcap2-bin

RUN groupadd -r linkerd && useradd -r -g linkerd linkerd
RUN setcap cap_net_raw,cap_net_admin+eip /usr/sbin/xtables-legacy-multi
RUN touch /run/xtables.lock && chmod 0666 /run/xtables.lock

USER linkerd

I am expecting the same result for proxy init container, but still to be verified.

But I am now also wondering if the powers are still same as before.

Run as Root with limited capabilities
vs
Run as Not Root, allow privilege escalation and same limited capabilities

My experience with linux dates from setuid days, then there was a gap when all the capabilities model evolved. And I have been an applications person all along, never a OS geek :-)

Plan to read more.

xynova · 2021-01-17T22:25:01Z

Following linkerd control plane PSP changes allow iptables to run as NON root
<   allowPrivilegeEscalation: false
---
>   allowPrivilegeEscalation: true
29c19
<     rule: RunAsAny
---
>     rule: MustRunAsNonRoot
my docker image where I have this working atm is
FROM ubuntu

RUN apt-get update && apt-get install -y --no-install-recommends iptables libcap2-bin

RUN groupadd -r linkerd && useradd -r -g linkerd linkerd
RUN setcap cap_net_raw,cap_net_admin+eip /usr/sbin/xtables-legacy-multi
RUN touch /run/xtables.lock && chmod 0666 /run/xtables.lock

USER linkerd
I am expecting the same result for proxy init container, but still to be verified.

But I am now also wondering if the powers are still same as before.

Run as Root with limited capabilities
vs
Run as Not Root, allow privilege escalation and same limited capabilities

My experience with linux dates from setuid days, then there was a gap when all the capabilities model evolved. And I have been an applications person all along, never a OS geek :-)

Plan to read more.

Running as non-root seems to be always a better choice. Granting root means the process gets write access to all files though and no capabilities prevent that from my experiments.

The allow privilege escalation is equivalent to docker run --security-opt=no-new-privileges. I am not entirely sure how the capabilites workflow happens in linux but it seems that granting capabilities does not mean that the process automatically has them when run. I have the feeling it still has to request them once trying to perform a privilege action and then this no-new-privileges prevents the process from getting anything on top of what it started with.

chrischdi · 2021-10-24T09:47:44Z

The changes at linkerd/linkerd2-proxy-init#49 lead to an up and running linkerd without the need of allowPrivilegeEscalation: true. The trick is to add the capabilities to the proxi-init binary too, because if it doesn't have them it cannot call a binary which requires them.

However, for now I am stuck at the injected init containers which are currently not allowed to run :-)

chrischdi · 2021-10-24T11:51:17Z

Requires for me the following changes in this repo for the controller:

diff --git a/charts/partials/templates/_proxy-init.tpl b/charts/partials/templates/_proxy-init.tpl
index 32411552..93e23a37 100644
--- a/charts/partials/templates/_proxy-init.tpl
+++ b/charts/partials/templates/_proxy-init.tpl
@@ -44,8 +44,6 @@ securityContext:
   privileged: false
   {{- end }}
   readOnlyRootFilesystem: true
-  runAsNonRoot: false
-  runAsUser: 0
 terminationMessagePolicy: FallbackToLogsOnError
 {{- if or (not .Values.cniEnabled) .Values.proxyInit.saMountPath }}
 volumeMounts:

And so I'm running completely as non-root :-)

Linkerd proxy-init container is currently enforced to run as root. Removes hardcoding `runAsNonRoot: false` and `runAsUser: 0`. This way the container inherits the user ID from the proxy-init image instead which may allow to run as non-root. Validation should happen during the usual tests. Fixes linkerd#5505 Signed-off-by: Schlotter, Christian <christian.schlotter@daimler.com>

Linkerd proxy-init container is currently enforced to run as root. Removes hardcoding `runAsNonRoot: false` and `runAsUser: 0`. This way the container inherits the user ID from the proxy-init image instead which may allow to run as non-root. Fixes #5505 Signed-off-by: Schlotter, Christian <christian.schlotter@daimler.com>

Linkerd proxy-init container is currently enforced to run as root. Removes hardcoding `runAsNonRoot: false` and `runAsUser: 0`. This way the container inherits the user ID from the proxy-init image instead which may allow to run as non-root. Fixes #5505 Signed-off-by: Schlotter, Christian <christian.schlotter@daimler.com> (cherry picked from commit 9853353) Signed-off-by: Oliver Gould <ver@buoyant.io>

adleong added the help wanted label Jan 11, 2021

alpeb mentioned this issue Jan 21, 2021

changes to run as non root, need to go hand in had with helm changes … linkerd/linkerd2-proxy-init#31

Closed

adleong assigned alpeb Oct 26, 2021

chrischdi mentioned this issue Oct 27, 2021

Allow proxy-init container to run as non-root #7162

Merged

alpeb closed this as completed in #7162 Nov 5, 2021

github-actions bot locked as resolved and limited conversation to collaborators Dec 6, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Linkerd without CNI - run as non-root #5505

Linkerd without CNI - run as non-root #5505

part-time-githubber commented Jan 11, 2021 •

edited

Loading

adleong commented Jan 11, 2021

part-time-githubber commented Jan 11, 2021

part-time-githubber commented Jan 12, 2021 •

edited

Loading

part-time-githubber commented Jan 13, 2021 •

edited

Loading

part-time-githubber commented Jan 13, 2021

wmorgan commented Jan 15, 2021

part-time-githubber commented Jan 16, 2021 •

edited

Loading

xynova commented Jan 17, 2021 •

edited

Loading

chrischdi commented Oct 24, 2021

chrischdi commented Oct 24, 2021

Linkerd without CNI - run as non-root #5505

Linkerd without CNI - run as non-root #5505

Comments

part-time-githubber commented Jan 11, 2021 • edited Loading

Feature Request

What problem are you trying to solve?

How should the problem be solved?

Any alternatives you've considered?

How would users interact with this feature?

adleong commented Jan 11, 2021

part-time-githubber commented Jan 11, 2021

part-time-githubber commented Jan 12, 2021 • edited Loading

part-time-githubber commented Jan 13, 2021 • edited Loading

part-time-githubber commented Jan 13, 2021

wmorgan commented Jan 15, 2021

part-time-githubber commented Jan 16, 2021 • edited Loading

xynova commented Jan 17, 2021 • edited Loading

chrischdi commented Oct 24, 2021

chrischdi commented Oct 24, 2021

part-time-githubber commented Jan 11, 2021 •

edited

Loading

part-time-githubber commented Jan 12, 2021 •

edited

Loading

part-time-githubber commented Jan 13, 2021 •

edited

Loading

part-time-githubber commented Jan 16, 2021 •

edited

Loading

xynova commented Jan 17, 2021 •

edited

Loading