SPLUNK_USER is root in universalforwarder

[kdthanvi]

##Details :
I want to run splunk-universalforwarder with non-root user. I checked the detailed docker file and from this I thought it should be user splunk.
But when I run the docker image and check using docker exec I see the user to be root (though everything inside is with splunk:splunk permission).

I am using splunk/universalforwarder:7.1.0 image.

To reproduce :

docker run -d  -p 9997:9997 -e 'SPLUNK_START_ARGS=--accept-license --seed-passwd changeme' splunk/universalforwarder:7.1.0

docker exec -it <container id> bash

OR

docker run -it  -p 9997:9997 -e 'SPLUNK_START_ARGS=--accept-license' -e 'SPLUNK_PASSWORD=splunk!!!NEW0' splunk/universalforwarder:7.1.0

# enter the password again when prompted
# After "Done", from another terminal

docker exec -it <container id> bash

You can see : root@<container id>$

Even in the documentation here it is written that user is splunk, but when I run it just runs with root.

Could you please help me here ? Thanks.

PS : I tried with splunk/universalforwarder:latest also and the results are same!

[outcoldman]

Just verified, seems like it is not only a UF problem but also Enterprise image as well.
Seems like the reason for that, because ansible uses sudo, see #36 (comment), and so launches splunk as root.

I exec in the wrong container. All worked as expected - I see splunk as a user for UF and Enterprise image.

[outcoldman]

@kdthanvi Just verified UF and Enterprise images 7.2.0, the actual UID of the processes are 999, and I see Splunk as a user running processes

[aegershman]

(Sorry for off-topic, but @outcoldman could I ask what that dashboard view is in the screenshot you posted? Is that from within Splunk? How did you get that data?)

[outcoldman]

Also just run docker run -it -e 'SPLUNK_START_ARGS=--accept-license' -e 'SPLUNK_PASSWORD=changeme' splunk/universalforwarder:latest

and got 999 as uid

splunk@5d5ee3e77e7b:/$ ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
splunk       1  0.1  0.0  18028  2928 pts/0    Ss+  15:27   0:00 /bin/bash /sbin/entrypoint.sh start-service
splunk     139  3.3  1.2 392600 79208 ?        Sl   15:27   0:00 splunkd -p 8089 start
splunk     140  0.0  0.2  80960 13156 ?        Ss   15:27   0:00 [splunkd pid=139] splunkd -p 8089 start [process-runner]
splunk     257  0.0  0.0   4232   652 pts/0    S+   15:27   0:00 tail -n 0 -f /opt/splunkforwarder/var/log/splunk/splunkd_stderr.log
splunk     258  1.0  0.0  18204  3276 pts/1    Ss   15:28   0:00 bash
splunk     266  0.0  0.0  36636  2796 pts/1    R+   15:28   0:00 ps aux

@kdthanvi could you verify again with latest? I assume 7.1.0 behaves differently, because it is an old/legacy image.

[outcoldman]

@aegershman this dashboard is part of our (https://www.outcoldsolutions.com) application Monitoring Docker: Metrics and Log Forwarding

[kdthanvi]

@outcoldman Thank you for the great help.
I deleted all the images I had regarding splunk, I did clean docker run. Looks like the image splunk/universalforwarder:latest works as you said (with docker exec also I got logged in as splunk user).

[kdthanvi]

When I use the latest image (with my production setup which is using kubernetes), it's NOT working.
Reason being :
Due to some constraints, I have kubernetes security policy set assecurityContext : runAsNonRoot : true (this is the reason I wanted to run the splunkforwarder as non-root user).
And when I am running the splunk/universalforwarder:latest image, I am getting the error Error: container has runAsNonRoot and image has non-numeric user (splunk), cannot verify user is non-root.

I checked the uid in my local container with id -u splunk & I can see it as 999.
But somehow to k8s it is seen as uid == nil (explained here).

@outcoldman any way I can resolve/debug this ?
Is there any document you've, where I can build the image locally (with this git repo) ?
PS : I tried building it but I am not sure about the arguments here.

Please let me know.
Thanks!

[outcoldman]

@kdthanvi I believe current image has issues with non-root user, because it uses sudo access sometimes #36
I am not 100% sure about that

[nwang92]

I don't think #36 is relevant here, this seems to be an implementation detail with Kubernetes specifically. I found kubernetes/kubernetes#59819 which seems to point out that in order for the securityContext/runAsNonRoot to work, the Dockerfile itself has to be built with USER <id> specified instead of USER <name>.

@kdthanvi can you confirm with Kubernetes version you're running in production? Additionally, you should be able to build your own image using the Makefile in this repository (ex. make splunk-debian-9). Also based on that github issue you linked, seems like you might be able to circumvent this by changing runAsNonRoot: true to runAsUser: 999. But that might not be applicable depending on your specific security requirements.

[kdthanvi]

Hi @nwang92 Thanks for the comment.
I thought the same when I read the error message, so I update the Dockerfile to have USER <id> instead of USER <name>.
The image runs well in my local (thanks to you, I could build successfully using make uf-debian-9).
But when I am running in the kubernetes cluster, the pod isn't starting as it should(it's is CrashLoopBackOff state) and the slpunk-uf container is giving me below error :

$ kubectl logs <pod-name> -c <container-name>
sudo: unable to change to root gid: Operation not permitted
sudo: unable to initialize policy plugin

Any idea on this ?

I am using kubernetes version v1.9.6 .

Also, about runAsUser : 999 this could help but there are more than 1 containers running in the same pod. So putting it use fixed user id for each container would not be a good way (I guess).
Also, as I mentioned there are security issues so I am not very sure if I could go for that.

Please let me know. Thanks!

[kdthanvi]

Please check this commit. This is how I changed/added to make the uf-debian-9 image run with USER <id>.

[kdthanvi]

I checked the other parts of the uf & theentrypoint.sh also has many sudo tasks. So I tried to change the usermod -aG sudo ${SPLUNK_USER} to usermod -aG sudo -u ${SPLUNK_USER_ID} ${SPLUNK_USER} as well.
But then the docker run -it -e 'SPLUNK_START_ARGS=--accept-license' -e 'SPLUNK_PASSWORD=changeme' splunkforwarder-debian-9 fails with too many permission denied errors.

[nwang92]

@kdthanvi gotcha, yeah I'm guessing changing ${SPLUNK_USER} to ${SPLUNK_USER_ID} will likely fail downstream. I suppose the areas of concern are mainly in the entrypoint in this particular case? Or other areas where privilege escalation comes into play?

I don't have a k8s cluster readily available for that particular version, but I'm going to try building the image from your commit and seeing if I can get the basic docker run command working.

[kdthanvi]

Hi @nwang92 just fyi, I did check the entrypoint and it's loaded with sudo. I did try to change(removed sudo and gave the splunk:9999 user permission for those directories) all those as well, but then I ended up facing many permission issues & No such file or directory issues.
Here is the error that I am getting during docker build.

Btw, with the commit I shared, the image builds successfully and even docker run works well in my local. But the error happens in k8s cluster, due to the entrypoint's sudo operations.

PS : Thanks for your support!

[matthewmodestino]

Hey kdthanvi, out of curiosity, what are you using the UF for? Sidecar for pods running in k8s? or to actually monitor k8s?

[kdthanvi]

@matthewmodestino I am running splunk-uf as a sidecar container in my k8s pod for getting the application logs (from the app container, running inside the same pod).

[nwang92]

@kdthanvi I built the image from your fork (https://hub.docker.com/r/nwang92/uf-k8s-test/) and tried it out but I'm not able to reproduce the issue you're seeing, but it's possible that I don't have the securityContext defined correctly. For reference, here's my pod yaml:

apiVersion: v1
kind: Pod
metadata:
  name: uf-seccontext
spec:
  securityContext:
    runAsNonRoot: true
  containers:
  - name: uf-seccontext
    image: nwang92/uf-k8s-test
    imagePullPolicy: Always
    env:
      - name: SPLUNK_START_ARGS
        value: --accept-license
      - name: SPLUNK_PASSWORD
        value: helloworld!
    securityContext:
      runAsNonRoot: true

My pod seems to be running ok:

uf-seccontext                             1/1       Running     0          8m

And the user/group ID's match what you've defined in that commit:

$ kubectl exec -it uf-seccontext bash
splunk@uf-seccontext:/$ id -u splunk
9999
splunk@uf-seccontext:/$ id -g splunk
9999

I am using k8s 1.8.4, so it's possible this may be a feature in a new version of the engine. But wondering if you have different securityContext definitions that I might have missing?

[kdthanvi]

Hi @nwang92 What we're doing is not in yaml, but in podsecuritypolicy.
In one line how we're managing is : podsecuritypolicy -> role -> rolebinding -> serviceaccount -> pod
So, maybe specifying this in yaml won't restrict the use of sudo as done in podsecuritypolicy (I am not sure about this though).

[nwang92]

Do you happen to know what your PodSecurityPolicy is set to? I tried creating a PodSecurityPolicy with runAsUser defined as below:

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: example
spec:
  privileged: false  # Don't allow privileged pods!
  # The rest fills in some required fields.
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  runAsUser:
    rule: MustRunAsNonRoot
  fsGroup:
    rule: RunAsAny
  volumes:
  - '*'

Using this security policy, I followed the examples from the docs and setup a user/role/role-binding:

kubectl create namespace psp-example
kubectl create serviceaccount -n psp-example fake-user
kubectl create rolebinding -n psp-example fake-editor --clusterrole=edit --serviceaccount=psp-example:fake-user

alias kubectl-admin='kubectl -n psp-example'
alias kubectl-user='kubectl --as=system:serviceaccount:psp-example:fake-user -n psp-example'

kubectl-admin create -f example-psp.yaml
kubectl-admin create role psp:unprivileged --verb=use --resource=podsecuritypolicy --resource-name=example
kubectl-admin create rolebinding fake-user:psp:unprivileged --role=psp:unprivileged --serviceaccount=psp-example:fake-user

But still unable to reproduce your exact issue, using the same pod yaml from above (nwang92/uf-k8s-test image).

$ kubectl-user create -f security-context.yml
pod/uf-seccontext created
$ kubectl-user get po
NAME                   READY   STATUS              RESTARTS   AGE
uf-seccontext          1/1     Running             0          1m

[lephino]

Is this issue good to close out?

[bekirby]

Any idea when this update will be released?

[mikedickey]

I believe the new 7.2.2 container published yesterday includes a fix for this.

[bekirby]

Thanks @mikedickey!