Node behind nginx, giving 502 Bad Gateway after a few hours

[henit]

• Node.js Version: 4.8.3
• OS: Ubuntu 16.04
• Express: 4.14.0
• MongoDB: 3.4.5
• PM2: 2.0.18

Express API application, running on Node behind Nginx. Have been running without problems for many months, then after a recent code update, it started giving 502 Bad Gateway. So probably caused by the code, but so far I have not been able to find the reason. This does NOT happen happen immediately after server start/restart, the API works without problems for a few hours, then it starts respoding with 502 until I restart the node application.

top-output the last time this happened (earlier today):

top - 07:54:23 up 4 days, 20:19,  3 users,  load average: 0.56, 0.59, 0.65
Tasks: 138 total,   1 running, 137 sleeping,   0 stopped,   0 zombie
%Cpu(s): 32.4 us,  6.3 sy,  0.0 ni, 58.7 id,  0.0 wa,  0.0 hi,  2.2 si,  0.3 st
KiB Mem :  4046688 total,  1000676 free,  1497976 used,  1548036 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  2191124 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                  

21946 henrik    20   0 1605992 468336  17872 S  64.8 11.6 112:20.10 node
23390 mongodb   20   0 5948448 776540  29740 S  16.3 19.2 303:20.86 mongod

Nothing in the pm2 error output (or standard output) from node at the time when it occurs, but when I try to run requests towards the express API, nginx error logs:

2017/09/04 07:47:05 [error] 27938#27938: *10145 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 77.40.144.162, server: api.example.com, request: "GET /v0/path/123 HTTP/2.0", upstream: "http://127.0.0.1:8091/v0/path", host: "api.example.com", referrer: "http://example.com/"
2017/09/04 07:47:05 [error] 27938#27938: *10145 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 77.40.144.162, server: api.example.com, request: "GET /v0/path/123 HTTP/2.0", upstream: "http://127.0.0.1:8091/v0/path", host: "api.example.com", referrer: "http://example.com/"

[NickNaso]

Could you post your NGINX configuration?
Usually I use this configuration on NGINX

# Set upstream for my-app
upstream my-app {
	server 127.0.0.1:5000;
}

server {
	listen 80;
	listen [::]:80;
	server_name default_server;

	# Set path for access_logs
  	access_log /var/log/nginx/access.my-app.log combined;

	# Set path for error logs
  	error_log /var/log/nginx/error.my-app.log notice;

	# If set to on, Nginx will issue log messages for every operation
  	# performed by the rewrite engine at the notice error level
  	# Default value off
 	rewrite_log on;

	# Settings for my-app service
	location / {
		# Header settings for application behind proxy
    		proxy_set_header Host $host;
   		# proxy_set_header X-NginX-Proxy true;
    		proxy_set_header X-Real-IP $remote_addr;
    		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    		proxy_set_header Upgrade $http_upgrade;
    		proxy_set_header Connection "upgrade";

		# Proxy pass settings
    		proxy_pass http://my-app/;

		# Proxy redirect settings
    		proxy_redirect off;

		# HTTP version settings
   		proxy_http_version 1.1;

		# Response buffering from proxied server default 1024m
    		proxy_max_temp_file_size 0;

		# Proxy cache bypass define conditions under the response will not be taken from cache
    		proxy_cache_bypass $http_upgrade;

	}
}

Usually you get 502 error when NGINX don't find an active service on the port you set on the uptream section. I hope to be helpful Nick

[henit]

@NickNaso I would look into that if I always got the 502, but it only occurs after the app have been running for a few hours, so that indicates the nginx config is working.

[henit]

Ok, I have now found the source if the problem. It was actually a memory leak in monk js, that was caused by the lack of caching collections in my application code.

Still, it should be noted that the node app was still running with high cpu usage when it stopped responding to requests, and with no error output from the node process. I would think a memory leak causing too much memory use would lead to some kind of error output that would make it easier to look for causes.

Anyways, problem found, thanks.

[shellberg]

The high CPU (and loss of performance leading to 502) you observed will have been the extra effort and multiple triggering of the garbage collection as the memory pressure on the heap increased to a critical point. You'd only get an OOM at the terminal point that an allocation failed (when GC could not cause any space sufficient for the request).... which eventually it would, if enough badness had not already occurred!

[justColbs]

@shellberg What would be considered high CPU before loss of performance occurs? 90%? 40%?

I get lots of 502s under heavy load. Not sure if the cause is memory limitation or CPU utilization.

[mcmc4519]

have same problem

[apan-liu]

• Node.js Version: 4.8.3
• OS: Ubuntu 16.04
• Express: 4.14.0
• MongoDB: 3.4.5
• PM2: 2.0.18

Express API application, running on Node behind Nginx. Have been running without problems for many months, then after a recent code update, it started giving 502 Bad Gateway. So probably caused by the code, but so far I have not been able to find the reason. This does NOT happen happen immediately after server start/restart, the API works without problems for a few hours, then it starts respoding with 502 until I restart the node application.

top-output the last time this happened (earlier today):

top - 07:54:23 up 4 days, 20:19,  3 users,  load average: 0.56, 0.59, 0.65
Tasks: 138 total,   1 running, 137 sleeping,   0 stopped,   0 zombie
%Cpu(s): 32.4 us,  6.3 sy,  0.0 ni, 58.7 id,  0.0 wa,  0.0 hi,  2.2 si,  0.3 st
KiB Mem :  4046688 total,  1000676 free,  1497976 used,  1548036 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  2191124 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                  

21946 henrik    20   0 1605992 468336  17872 S  64.8 11.6 112:20.10 node
23390 mongodb   20   0 5948448 776540  29740 S  16.3 19.2 303:20.86 mongod

Nothing in the pm2 error output (or standard output) from node at the time when it occurs, but when I try to run requests towards the express API, nginx error logs:

2017/09/04 07:47:05 [error] 27938#27938: *10145 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 77.40.144.162, server: api.example.com, request: "GET /v0/path/123 HTTP/2.0", upstream: "http://127.0.0.1:8091/v0/path", host: "api.example.com", referrer: "http://example.com/"
2017/09/04 07:47:05 [error] 27938#27938: *10145 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 77.40.144.162, server: api.example.com, request: "GET /v0/path/123 HTTP/2.0", upstream: "http://127.0.0.1:8091/v0/path", host: "api.example.com", referrer: "http://example.com/"

Hello, have you solved the problem now ？

[andrehsmendes]

having the same problem

[g00nix]

We are getting the same problem with nginx, but independent of nodejs. After a few hours, nginx starts returning 502 and requires a restart.

[snavaneeth78]

Same issue here, specifically with Next.js

[bizaybsnt]

Has it got solved yet? I am also facing issue on Next.JS

[jayachandra9]

any solution for above issue @NickNaso

[avivkeller]

Hey, this issue is closed. If your experiencing a separate issue, please open a new issue