reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response #1092
Comments
yantaowu
added
status/need-triage
violetagg
removed
the
status/need-triage
|
@yantaowu There might be several causes about this. Can you confirm that you do not have these issues: |
violetagg
added
the
for/user-attention
Thanks a lot. I will try my best to obtain the tcp dump and have a feedback. |
|
@yantaowu I'm closing this. We can reopen it if it is needed. |
|
same problem |
|
@iceqing Did you check #1092 (comment)? If nothing from the links helps then create a new issue with a reproducible example. |
|
@violetagg I obtain the tcp dump，if we can check the connection state before sending request，maybe resolve the problem。 2020-06-04 06:58:09.151 WARN [reactor-http-epoll-8] [HttpClientConnect] [id: 0x483deb13, L:/172.18.206.6:39284 ! R:172.18.207.2/172.18.207.2:8151] The connection observed an error
|
|
@yantaowu So the one with port 8151 is that the server? Why this RST is sent? |
|
@violetagg The one with port 8151 is server. According to the dump, I think the server close the connection first, then the client with port 39284 send a request to the server, since the connection is closed, so return the RST to client. |
|
@yantaowu Why has the server closed the connection? Are there any logs on the server? |
|
@violetagg There are no logs on the server. I have a question, why sending a request to the server on the closed connection. Could we check the connection state before sending the request? |
|
@violetagg The post request with path "/route" is the request to the server, but the connection has been closed before sending at this time. |
|
when I made poor server that send response header only(not response body) so I think that PrematureCloseException is not always a reactor-netty problem. |
|
@yantaowu The connection state is checked several times before issuing the request.
So it is important to understand the scenario when this happens and why the server closed the connection. |
Thank you very much. |
|
+1 |
|
@ehongyu And you checked #1092 (comment) and it is nothing from the list. Can you take a tcp dump? I would really appreciate if you start a new issue related to your use case. It might seems similar but it might appear that it is a different root cause. |
|
I wish I could get a tcpdump, but this annoying issue only happened once in a while on the production box and our company's production box is beyond our reach. |
|
I get a tcpdump. client(10.157.122.166):
server(10.138.7.86): |
|
@iceqing So do you have some logs on the server that indicate why the server closes the connection? |
|
From the linked issue spring-cloud/spring-cloud-gateway#1148 It appears that the server is Tomcat with configurations and These two configurations might be a cause for a connection close. I would recommend to configure |
violetagg
added
status/invalid
|
2023-09-16 16:47:50.180 INFO 244 --- [ctor-http-nio-4] com.nari.amap.filter.AmapGatewayFilter : 输出请求接口路径：/amap-gateway-service/amap-app-service/doc.html java.lang.NoSuchMethodError: org.springframework.http.server.reactive.ServerHttpResponse.setRawStatusCode(Ljava/lang/Integer;)Z 2023-09-16 16:47:50.227 WARN 244 --- [ctor-http-nio-4] r.netty.http.client.HttpClientConnect : [id: 0x0c21051a, L:0.0.0.0/0.0.0.0:58395 ! R:/172.29.136.91:12000] The connection observed an error reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response 2023-09-16 16:47:50.236 ERROR 244 --- [ctor-http-nio-4] a.w.r.e.AbstractErrorWebExceptionHandler : [e392ca95] 500 Server Error for HTTP GET "/amap-gateway-service/amap-app-service/doc.html" reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response |
I use spring cloud gateway as our gateway, this exception occurs in our production environment randomly when the request arrived. I search the related issue on the github and found it had been fixed, but my reactor-netty is the latest release(0.9.7).
Steps to Reproduce
It is very difficult to reproduce，which only occurs in production environment randomly.
This is the log when the exception occurs:
2020-05-12 17:31:22.948 DEBUG [reactor-http-epoll-11] [PooledConnectionProvider] [id: 0xcbc4c890, L:/172.18.206.6:23016 - R:172.18.207.3/172.18.207.3:8151] Channel acquired, now 1 active connections and 0 inactive connections
2020-05-12 17:31:22.948 DEBUG [reactor-http-epoll-11] [HttpClientConnect] [id: 0xcbc4c890, L:/172.18.206.6:23016 - R:172.18.207.3/172.18.207.3:8151] Handler is being applied: {uri=http://172.18.207.3:8151/route, method=POST}
2020-05-12 17:31:22.948 DEBUG [reactor-http-epoll-11] [PooledConnectionProvider] [id: 0xcbc4c890, L:/172.18.206.6:23016 - R:172.18.207.3/172.18.207.3:8151] onStateChange(POST{uri=/route, connection=PooledConnection{channel=[id: 0xcbc4c890, L:/172.18.206.6:23016 - R:172.18.207.3/172.18.207.3:8151]}}, [request_prepared])
2020-05-12 17:31:22.948 DEBUG [reactor-http-epoll-11] [PooledConnectionProvider] [id: 0xcbc4c890, L:/172.18.206.6:23016 - R:172.18.207.3/172.18.207.3:8151] onStateChange(POST{uri=/route, connection=PooledConnection{channel=[id: 0xcbc4c890, L:/172.18.206.6:23016 - R:172.18.207.3/172.18.207.3:8151]}}, [request_sent])
2020-05-12 17:31:22.948 DEBUG [reactor-http-epoll-11] [PooledConnectionProvider] [id: 0xcbc4c890, L:/172.18.206.6:23016 ! R:172.18.207.3/172.18.207.3:8151] Channel closed, now 0 active connections and 0 inactive connections
2020-05-12 17:31:22.948 DEBUG [reactor-http-epoll-11] [PooledConnectionProvider] [id: 0xcbc4c890, L:/172.18.206.6:23016 ! R:172.18.207.3/172.18.207.3:8151] onStateChange(POST{uri=/route, connection=PooledConnection{channel=[id: 0xcbc4c890, L:/172.18.206.6:23016 ! R:172.18.207.3/172.18.207.3:8151]}}, [response_incomplete])
2020-05-12 17:31:22.949 WARN [reactor-http-epoll-11] [HttpClientConnect] [id: 0xcbc4c890, L:/172.18.206.6:23016 ! R:172.18.207.3/172.18.207.3:8151] The connection observed an error
reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
The configuration of spring cloud gateway:
spring.cloud.gateway.httpclient.pool.type=fixed
spring.cloud.gateway.httpclient.pool.max-connections=1000
spring.cloud.gateway.httpclient.pool.acquire-timeout=7500
spring.cloud.gateway.httpclient.pool.max-idle-time=90s
spring.cloud.gateway.httpclient.pool.max-life-time=90s
Your Environment
javar -version): jdk 1.8uname -a): centos 6.7The text was updated successfully, but these errors were encountered: