Kubeapps container fails with nginx error

[LP0101]

When deploying Kubeapps, I get the following nginx error:

Welcome to the Bitnami nginx container
Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-nginx
Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-nginx/issues
Send us your feedback at containers@bitnami.com

INFO  ==> ** Starting NGINX setup **
INFO  ==> Validating settings in NGINX_* env vars...
ERROR ==> The NGINX_DAEMON_USER environment variable cannot be empty when running as root

I'm using the latest version of the chart. This error happens no matter how I try to deploy it, even with vanilla configuration.

the kubeapps-internal-dashboard pods have the same error

Edit: I tried deploying version 1.3 of the chart, I'm getting the same issue. We weren't having this issue a week ago when it was last deployed, so I have no idea what could have changed.

[andresmgot]

hi @LP0101,

That is due to an update in the Nginx container. We have updated it to a new major version (1.6). This version ensures that if the container is run as root, the user specifies the environment variable NGINX_DAEMON_USER to something different than root as a security measure. By default the user used in the container should be 1001 so that shouldn't be a problem. Does your cluster has any policy that changes the default running user?

[LP0101]

We don't have any policies in place that affect user switching.

However, if I deploy the bitnami/nginx image manually, kubectl run nginx-test --image=bitnami/nginx, it runs just fine. The error only seems to occur with the kubeapps helm deployment.

Update: kubectl run nginx-test --image=bitnami/nginx:1.16.0-r3 fails with the same error. However, using the latest image continues to work. I'm not sure what's changed between them, but something about the 1.16.0-r3 tag isn't working properly.

[LP0101]

I got it deployed by setting the following values:

--set frontend.image.tag=latest --set dashboard.image.tag=latest

Not sure what caused the issue, but for now this fixed it

[andresmgot]

mm, that's weird. Let me ask the team maintaining the image to check if something changes. I am using 1.16.0-r3 and I don't see that issue. Can you confirm if it works for you with the tag 1.16.0-r28? Maybe the latest tag is cached in your cluster and it's using an older version.

[LP0101]

Running 1.16.0-r28 results in the same error.

This is getting really weird now, though. Running the images with both the latest and 1.16.0-r28 tags, I get the same sha hash with kubectl describe (32fc9cf8ff4dc38615113899619844a0395618957c7f1afc0b8f9fa7c6e06d47). However, latest spins up properly, while 1.16.0-r28 throws the nginx error.

I've attached the logs below so you can see this behavious for yourself. I'm really at a loss about how to explain it.

https://pastebin.com/tb044rtH

[juan131]

Hi @andresmgot @LP0101

I was unable to reproduce the issue:

$ kubectl run nginx-test --image=bitnami/nginx:1.16.0-r28
$ kubectl logs  $(kubectl get pod -l run=nginx-test -o jsonpath='{.items[0].metadata.name}')

Welcome to the Bitnami nginx container
Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-nginx
Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-nginx/issues
Send us your feedback at containers@bitnami.com

INFO  ==> ** Starting NGINX setup **
INFO  ==> Validating settings in NGINX_* env vars...
INFO  ==> Initializing NGINX...
INFO  ==> ** NGINX setup finished! **

INFO  ==> ** Starting NGINX **

The error you're facing is due to the validation below:

https://github.com/bitnami/bitnami-docker-nginx/blob/master/1.16/debian-9/rootfs/libnginx.sh#L160

When running the container as 'root', you need to provide the Nginx user/daemon using the env. variables NGINX_DAEMON_USER/NGINX_DAEMON_GROUP

By default this container runs as a non-root user, see https://github.com/bitnami/bitnami-docker-nginx/blob/master/1.16/debian-9/Dockerfile#L32
However, I guess you may have some securityPolicy defined on your k8s cluster that is forcing the container to be run as root user (ID=0). Could you try using the command below?

$ kubectl run --env="NGINX_DAEMON_USER=nginx" --env="NGINX_DAEMON_GROUP=nginx" nginx-test --image=bitnami/nginx:1.16.0-r28

[LP0101]

We don't have any sort of policy in the cluster that prevents running as non-root user, as we have several docker images which also switch user, just as the nginx one does.

Again, just to reiterate, the same image works as expected when running with the latest tag, but not when running with the 1.16.0-r28 tag. By which I mean, the user is properly being switched.

Running that last command, this is the output I get:

Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-nginx
Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-nginx/issues
Send us your feedback at containers@bitnami.com

INFO  ==> ** Starting NGINX setup **
INFO  ==> Validating settings in NGINX_* env vars...
INFO  ==> Initializing NGINX...
nginx:x:1000:
INFO  ==> ** NGINX setup finished! **

INFO  ==> ** Starting NGINX **
nginx: [alert] could not open error log file: open() "/opt/bitnami/nginx/logs/error.log" failed (13: Permission denied)
2019/05/30 15:06:32 [emerg] 1#0: unexpected end of parameter, expecting ";" in command lin

Update: r-29 works properly.

[andresmgot]

Update: r-29 works properly.

Does that mean that it works for you if you set the env var NGINX_DAEMON_USER and NGINX_DAEMON_GROUP?

Another thing that you can try, when installing the kubeapps chart is to set the values --set securityContext.enable=true --set securityContext.fsGroup=1001 that should force a securityContext to use a different user than root.

[LP0101]

Does that mean that it works for you if you set the env var NGINX_DAEMON_USER and NGINX_DAEMON_GROUP?

Nope, I can deploy the r-29 image without any env vars being set. Just works out of the box as expected.

Another thing that you can try, when installing the kubeapps chart is to set the values --set securityContext.enable=true --set securityContext.fsGroup=1001 that should force a securityContext to use a different user than root.

Ran this with the vanilla chart, so pulling the hardcoded images in the original values.yaml. Still results in the same error with the nginx container starting as root.

[prydonius]

So do we just need to update the chart to point to -r29?

[migmartri]

Was this a regression added in r28? What changed in r29 that makes it work now?

[andresmgot]

What changed in r29 that makes it work now?

There is no public change in the nginx repository so I don't know if that is really the issue. Since we are not able to reproduce it I have updated the image in the chart to r40 hoping that helps until we get more information. The changes will be available in the next release (that is in progress).

Maybe @juan131 can clarify if something changed in the image or in the base image that could affect that?

[juan131]

I don't see any relevant change on the latest revisions @andresmgot
And.. As you mentioned, I'm not able to reproduce the issue either.

[andresmgot]

@LP0101 can you confirm if the issue is still present?

[LP0101]

We've been wracking our brains about this for the past week, but recently stumbled upon this: kubernetes/kubernetes#78308

Turns out it wasn't an issue with kubeapps at all, but a general problem with Kubernetes, which explains the weird behaviour when using different tags.

Closing this issue.

[prydonius]

oh wow, great find @LP0101 - thanks for the update!

[jessehu]

Thanks for the findings. I came across the same issue after deploying the latest bitnami/nginx chart successfully using Kubeapps. My env is minikube 1.1.0 (Kubernetes v1.14.2) which has the bug kubernetes/kubernetes#78308. minikube 1.2.0 (Kubernetes v1.15.0) should work.