Nginx access logs are not showing up in stdout

[kirs]

What keywords did you search in NGINX Ingress controller issues before filing this one? logs, access_log, /dev/stdout

---

Is this a BUG REPORT or FEATURE REQUEST?: bug report

NGINX Ingress controller version: 0.19.0 (git-05025d6); nginx/1.15.3

Kubernetes version (use kubectl version): v1.9.7-gke.6, v1.10.0, v1.10.6-gke.2, v1.10.7-gke.2

Environment:

• Cloud provider or hardware configuration: GKE and minikube on macOS

What happened:

Nginx logs are not reported to stdout of nginx-ingress-controller container and therefore are not visible from kubectl logs.

What you expected to happen:

kubectl logs for nginx-ingress-controller pod should include nginx access_log, given that it's access_log is enabled by default

How to reproduce it (as minimally and precisely as possible):

kubectl apply -f deploy/mandatory.yaml
kubectl apply -f deploy/cloud-generic.yaml

$ kubectl -n ingress-nginx get pods
NAME                                       READY     STATUS    RESTARTS   AGE
default-http-backend-7fb86fb47d-ptmr8      1/1       Running   0          22m
nginx-ingress-controller-cccc694cd-j94vb   1/1       Running   0          5m

$ kubectl -n ingress-nginx logs -f nginx-ingress-controller-cccc694cd-j94vb
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:    0.19.0
  Build:      git-05025d6
  Repository: https://github.com/kubernetes/ingress-nginx.git
-------------------------------------------------------------------------------

nginx version: nginx/1.15.3
W0930 13:45:14.986665       7 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might no
t work.
I0930 13:45:14.987024       7 main.go:191] Creating API client for https://10.47.240.1:443
I0930 13:45:14.998289       7 main.go:235] Running in Kubernetes cluster version v1.10+ (v1.10.6-gke.2) - git (clean) commit 384b4eaa132ca9a295
fcb3e5dfc74062b257e7df - platform linux/amd64
I0930 13:45:15.000640       7 main.go:100] Validated ingress-nginx/default-http-backend as the default backend.
I0930 13:45:15.318367       7 nginx.go:256] Starting NGINX Ingress controller
I0930 13:45:15.357240       7 event.go:221] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"udp-services", UID:"993
037c4-c4b6-11e8-8c41-42010a800128", APIVersion:"v1", ResourceVersion:"1082", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-
nginx/udp-services
I0930 13:45:15.357527       7 event.go:221] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"nginx-configuration", U
ID:"98c4ce35-c4b6-11e8-8c41-42010a800128", APIVersion:"v1", ResourceVersion:"1080", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap i
ngress-nginx/nginx-configuration
I0930 13:45:15.357613       7 event.go:221] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"tcp-services", UID:"98f
a8d81-c4b6-11e8-8c41-42010a800128", APIVersion:"v1", ResourceVersion:"1081", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-
nginx/tcp-services
I0930 13:45:16.519219       7 nginx.go:277] Starting NGINX process
I0930 13:45:16.519577       7 leaderelection.go:185] attempting to acquire leader lease  ingress-nginx/ingress-controller-leader-nginx...
I0930 13:45:16.521674       7 controller.go:171] Configuration changes detected, backend reload required.
I0930 13:45:16.532274       7 leaderelection.go:194] successfully acquired lease ingress-nginx/ingress-controller-leader-nginx
I0930 13:45:16.532650       7 status.go:197] new leader elected: nginx-ingress-controller-5b6864749-vrmvd
I0930 13:45:16.654705       7 controller.go:187] Backend successfully reloaded.
I0930 13:45:16.655577       7 controller.go:197] Initial synchronization of the NGINX configuration.
I0930 13:45:17.659946       7 controller.go:204] Dynamic reconfiguration succeeded.
<... aren't access logs supposed to be here? ...>

After doing a few requests to the service with curl, I don't see any nginx logs. I'm using default configuration where nginx.ingress.kubernetes.io/enable-access-log is supposed to be true by default.

Checking if access_log is enabled in nginx.conf:

$ kubectl -n ingress-nginx exec nginx-ingress-controller-cccc694cd-j94vb cat /etc/nginx/nginx.conf | grep access_log
        access_log /var/log/nginx/access.log upstreaminfo if=$loggable;
...

access_log is enabled. Where does it point to?

$ kubectl -n ingress-nginx exec nginx-ingress-controller-cccc694cd-j94vb -- ls -la /var/log/nginx
total 8
drwxr-xr-x 1 www-data www-data 4096 Aug 12 02:06 .
drwxr-xr-x 1 www-data www-data 4096 Aug 12 02:06 ..
lrwxrwxrwx 1 root     root       11 Aug 12 02:06 access.log -> /dev/stdout
lrwxrwxrwx 1 root     root       11 Aug 12 02:06 error.log -> /dev/stderr

It's linked to /dev/stdout, so it should appear in container logs which capture stdout.

Anything else we need to know:

Why do I think that access_log is supposed to be in container's stdout? See this gist with logs reported by another user.

Steps I took to try to pin this:

• ingress-nginx versions tried:

  • 0.19.0
  • 0.18.0
  • 0.17.1

• providers tried:

  • GKE
    • k8s 1.10.6-gke.2
    • k8s 1.10.7-gke.2
    • k8s v1.9.7-gke.6
  • minikube
    • k8s v1.10.0

• tried to ssh to the node directly and run docker logs, no access logs there either

[aledbf]

@kirs how are you testing this?
Please execute something like
kubectl exec -n ingress-nginx nginx-ingress-controller-cccc694cd-j94vb curl localhost and check the logs

[kirs]

I was testing this by finding the IP with kubectl -n ingress-nginx get service and then hitting that IP with curl.

kubectl exec <pod> curl localhost doesn't make the controller pod to produce any logs either.

In both cases I get default backend - 404 as a response from default-http-backend pod, which is the expected behaviour.

[ElvinEfendi]

@kirs Access log is disabled for default server (the 404 server) (you can see that by inspecting the generated Nginx configuration, search for access_log off;). Try creating a dummy ingress and execute the same curl command with -H "Host: <the host you configured in your domain>". You can also create a catch all ingress and then execute the same curl command you will see logs.

[kirs]

Oh, thanks a lot, I've spent like 4 hours trying to find the issue 🤦‍♂️

It works now 🎉 Do you mind if I add a sample ingress to the dev setup (make dev-env) that would point to default-http-backend? IMO it would help new developers to start playing with the controller and to see logs.

[ElvinEfendi]

Do you mind if I add a sample ingress to the dev setup (make dev-env) that would point to default-http-backend

@kirs I'd think catch all server is a special beast that only minority of ingress-nginx users need/use. I'd instead create an ingress with hostname such as ingress-nginx.dev or something similar pointing to docs//examples/http-svc.yaml instead of default-http-backend. We can then update the documentation accordingly (i.e talk about curl -H "Host: ingress-nginx.dev").

[aledbf]

Closing. We already have an example to test the ingress controller

[mentgmery]

@kirs Access log is disabled for default server (the 404 server) (you can see that by inspecting the generated Nginx configuration, search for access_log off;). Try creating a dummy ingress and execute the same curl command with -H "Host: <the host you configured in your domain>". You can also create a catch all ingress and then execute the same curl command you will see logs.

how to create the dummy ingress? I still can't understand. same issue for me, can't see any logs if tail stdout and stderr.

[mentgmery]

Closing. We already have an example to test the ingress controller

I dont understand teh relationship between the example and the stdout and stderr logs? could you help me, thanks:)

[Orabig]

@aledbf closing comment is not helping : I'm in the same situation as @mentgmery : I have the same issue than OP, and the closing message talks about "an example to test the ingress controller" which is not related at all.
I would like to be able to see the nginx logs, and I can't find it (and this ticket IS right : it should be present in stdout, and it's not)

[mentgmery]

@Orabig finally I found the root cause of the problems. even it outputs logs to stdout, it still save the data in the disk. at that time, when i looked into these logs(/var/log/containers), i found logs blank, that was because my docker version is too low, and it was docker's bug to not output the logs. after i upgrade the docker to 18.03, then all works. hope, it can help you:) and i recommend the efk strongly, pls have a look.

[johnraz]

For people ending up here like myself:

There is a configmap value that can be set to enable access logs on the default backend, see here: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#enable-access-log-for-default-backend