Nginx doesn't work after restarting upstream pod without restarting nginx itself.

[khteh]

Please run microk8s.inspect and attach the generated tarball to this issue.

We appreciate your feedback. Thank you for using microk8s.
My Nginx connects to upstream headless service which serves statefulset webapi. I restart both the statefulset and the headless service without the nginx and it results in 502 Bad Gateway. I have to restart nginx to get the whole to work again. Is this the expected behaviour?

[ktsakalozos]

Hi @khteh ,

Can you share the yaml manifests needed for your deployment as well as instructions on how to reproduce the issue?

Thanks

[khteh]

Downstream: NodePort Service (svc-frontend) > Nginx Deployment (nginx)
Upstream: Headless Service (svc-web) > Statefulset tomcat application (sts-web)
kubectl get all (Note the cluster IP)
curl http:///webapi/webmethod (OK)

In no particular order:
(1) kubectl delete svc-web
(2) kubectl delete sts-web

then,
(1) kubectl apply -f svc-web.yml
(2) kubectl apply -f sts-web.yml

kubectl get all (Get the clusterIP which doesn't change because the nginx is still running).
curl http:///webapi/webmethod (502 Bad Gateway)

I have to do the following to resume the system which I am not sure is normal behaviour:
kubectl delete svc-frontend
kubectl delete nginx
kubectl apply -f svc-frontend.yml
kubectl apply -f nginx.yml

kubectl get all (Note the cluster IP)
curl http:///webapi/webmethod (OK)

[andriy-f]

@khteh It may be related to the fact that nginx when defining uptream address in proxy_pass config directive, nignx resolves dns name to ip only once. And if upstream ip address changes, he don't re-resolve it automatically.

[khteh]

@andriy-f, sounds sensible. Is this a bug that needs to be fixed that changes to the upstream need to be propagated to the downstream? In this case, the detach / attach of upstream svc/pod to the nginx need to be know to nginx and trigger it to refresh itself. Simple observer pattern.

[khteh]

Restarting nginx will cause service downtime to a production environment.

[andriy-f]

@khteh This should not cause downtime nginx -s reload https://nginx.org/en/docs/beginners_guide.html#control

[khteh]

How to use this with the nginx running in k8s cluster?

[andriy-f]

Run as separate process alongside main nginx (hint: supervisord) which will trigger nginx -s ...

[khteh]

How does it fit into the k8s cluster architecture? Does it need a separate deployment / pod? Just for that purpose?

[khteh]

With reference to https://serverfault.com/questions/240476/how-to-force-nginx-to-resolve-dns-of-a-dynamic-hostname-everytime-when-doing-p/946404#946404, I change my nginx configuration accordingly but still doesn't work:

        listen 80;
        access_log /var/log/nginx/app.access_log main;
        error_log /var/log/nginx/app.error_log;
        resolver 127.0.0.1 [::1]:5353 valid=10s;
        set $upstream http://svc-web;
        location / {
          proxy_pass $upstream;
          proxy_http_version 1.1;
        }
      }

What is the right resolver address that should be used?

[frankgululu]

@andriy-f I denfined svc name in Nginx configuration, instead of ip addr. it still can't work(need manually reboot Nginx after upstream server was restarted)

location /api/ {
    proxy_pass http://$svc.$namespace:8081/;
}

}

[andriy-f]

@frankgululu did you set resolver?

[frankgululu]

There have no manual setting...Kubeneters Service discovery can parse it...
The Nginx still work until the upstream server(Pod) was reboot.....And it works again after reboot Nginx