balancer_by_lua_block : Reusing stale connection

[rohitjoshi]

Hello,
I am using balancer_by_lua for upstream routing which works great. We are seeing intermittent errors where nginx is caching the connection while upstream has already terminated.

See the below image. xxx.xxx.xxx.159 is OpenResty server and xxx.xxx.xxx.27 is upstream server.

Thanks,
Rohit Joshi

[agentzh]

@rohitjoshi What's in your nginx error logs (assuming you are not setting the error log filter level too high)?

Usually such RSTs should be harmless since nginx will automatically retry a new connection for the current request.

[agentzh]

@rohitjoshi The nginx core does check broken connections in the connection pool, but race conditions are inevitable in case the nginx just tries to use an in-pool connection before it can handle a read event for the connection termination request from the remote.

[rohitjoshi]

The issue is due to AWS NAT gateway terminates connection after 5 minutes.

I am seeing below error in the error log.
' upstream prematurely closed connection while reading response header from upstream'
Nginx doesn't retry automatically. I had to enable next upstream on 502 for post requests.

[agentzh]

@rohitjoshi Yes, you need to enable the next upstream mechanism.

[thibaultcha]

Closing this and considering it answered. Latest Nginx versions also have keepalive_timeout and keepalive_requests to remedy such race-conditions.