Connection prematurely closed BEFORE response

[FangPengbo]

Describe the bug

@spencergibb
Hi
Version information:

springboot:2.3.1.RELEASE
spring-cloud-starter-gateway:2.2.6.RELEASE
reactor-netty:0.9.8.RELEASE
reactor-core:3.3.6.RELEASE
<spring.cloud.version>Hoxton.SR9</spring.cloud.version>

configuration information:

  cloud:
    gateway:
      httpclient:
        pool:
          type: fixed
          max-connections: 1000
          acquire-timeout: 80000
          max-life-time: 300s
          max-idle-time: 60s
        connect-timeout: 8000
        response-timeout: 60s

Stack information

The first：

2020-11-18 13:50:19.550 [TID: N/A] ERROR [org.springframework.boot.autoconfigure.web.reactive.error.AbstractErrorWebExceptionHandler] [daad6f08-2733]  500 Server Error for HTTP POST "/13/sinoiacav83/caserver"
reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	|_ checkpoint ? org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
	|_ checkpoint ? org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
	|_ checkpoint ? HTTP POST "/13/sinoiacav83/caserver" [ExceptionHandlingWebHandler]
Stack trace:

The second：

2020-11-18 13:59:04.581 [TID: N/A] ERROR [org.springframework.boot.autoconfigure.web.reactive.error.AbstractErrorWebExceptionHandler] [c0a9ea32-3007]  500 Server Error for HTTP POST "/mobile2dev/api/send?sign=7eb5c829a40d6528477a2163aab58f40"
io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: msg.umeng.com/203.119.211.232:80
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	|_ checkpoint ? org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
	|_ checkpoint ? org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
	|_ checkpoint ? HTTP POST "/mobile2dev/api/send?sign=7eb5c829a40d6528477a2163aab58f40" [ExceptionHandlingWebHandler]
Stack trace:
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
	at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124) ~[netty-transport-native-unix-common-4.1.50.Final.jar!/:4.1.50.Final]
	at io.netty.channel.unix.Socket.finishConnect(Socket.java:243) ~[netty-transport-native-unix-common-4.1.50.Final.jar!/:4.1.50.Final]
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:672) [netty-transport-native-epoll-4.1.50.Final-linux-x86_64.jar!/:4.1.50.Final]
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:649) [netty-transport-native-epoll-4.1.50.Final-linux-x86_64.jar!/:4.1.50.Final]
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:529) [netty-transport-native-epoll-4.1.50.Final-linux-x86_64.jar!/:4.1.50.Final]
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:465) [netty-transport-native-epoll-4.1.50.Final-linux-x86_64.jar!/:4.1.50.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [netty-transport-native-epoll-4.1.50.Final-linux-x86_64.jar!/:4.1.50.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.50.Final.jar!/:4.1.50.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.50.Final.jar!/:4.1.50.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.50.Final.jar!/:4.1.50.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]

These two kinds of errors occur every day。

Sample
We are using a way to load routes dynamically

    public void addRoute(RouteDefinition definition) throws BussinessException{
        try{
            routeDefinitionWriter.save(Mono.just(definition)).subscribe();
            this.publisher.publishEvent(new RefreshRoutesEvent(this));
        }catch(Exception e){
            throw new BussinessException("add route fail: "+definition.getId()+"，异常信息："+e);
        }
    }

I've seen other suggestions
We tried a lot of solutions in isseus
It doesn't seem to work

Please help us!
Thanks!

[ctlove0523]

@Fangfeikun
see this issue from reactor-netty issue1375

[Owenxh]

Specify the http client pool max idle time less than 30s

spring.cloud.gateway.httpclient.pool.max-idle-time = 10s

We found the problem before, since we changed the previous config, it never happened again.

[ctlove0523]

@Owenxh
thanks for your suggestion.when i configure idle time equals 10s,but if has more request in per second like 2000request/s this problem occur again,so i think reactor-netty project need to work hard for fix this。

[FangPengbo]

@Owenxh
thanks for your suggestion.when i configure idle time equals 10s,but if has more request in per second like 2000request/s this problem occur again,so i think reactor-netty project need to work hard for fix this。

@ctlove0523
@Owenxh
Will you still show up after you try？
I'll check it out and wait for my message

[FangPengbo]

We used to have this configuration
acquire-timeout: 8000
When I set the parameter to the bottom like this
acquire-timeout: 80000
There are no problems in our test environment today
I will continue to observe tomorrow
Do you have a recommended value for this parameter?
Our QPS is 1000 / s

[FangPengbo]

2020-11-20 16:45:01.611 [TID: N/A] ERROR [org.springframework.boot.autoconfigure.web.reactive.error.AbstractErrorWebExceptionHandler] [22831776-15299]  500 Server Error for HTTP POST "/3302/sinoiacsv20/csserver"
reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
|_ checkpoint ? org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
|_ checkpoint ? org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
|_ checkpoint ? HTTP POST "/3302/sinoiacsv20/csserver" [ExceptionHandlingWebHandler]
Stack trace:
2020-11-20 16:45:02.378 [TID: N/A] ERROR [org.springframework.boot.autoconfigure.web.reactive.error.AbstractErrorWebExceptionHandler] [9d3ca003-15297]  500 Server Error for HTTP POST "/3302/sinoiacsv20/csserver"
reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
|_ checkpoint ? org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
|_ checkpoint ? org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
|_ checkpoint ? HTTP POST "/3302/sinoiacsv20/csserver" [ExceptionHandlingWebHandler]
Stack trace:

@ctlove0523 @Owenxh
@spencergibb
After modifying the parameters, an exception is still thrown

[ctlove0523]

@FangPengbo when you reproduce this exception can you make a tcpdump file for analysis.

[AsCat]

@FangPengbo
Can you help test it with this PR ?

[hpeimer]

I have tried all the advice but have been unable to prevent "Connection prematurely closed BEFORE response".
I did find a solution which provides a good patch, by enabling retry in the Gateway:

spring:
  cloud:
    gateway:
      httpclient:
        pool:
          max-idle-time: 2000
      default-filters:
        - name: Retry
          args:
            methods: GET,PUT,POST,DELETE
            exceptions:
              - reactor.netty.http.client.PrematureCloseException

[ben-bennur-thg]

This error went away for me after downgrading the dev.akkinoc.spring.boot dependency I have to 3.4.1 or completely removing it