reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response，springboot2.1.6 #1148

kinsey-jian · 2019-07-03T12:32:03Z

springboot2.1.6.RELEASE and Spring Cloud Greenwich.SR1

application.yml

spring:
application:
name: gateway-service
cloud:
gateway:
default-filters:
- StripPrefix=1
routes:
- id: provider_api_route
uri: lb://provider-service
predicates:
- Path=/web/**
discovery:
locator:
enabled: true
consul:
enabled: true
discovery:
instanceId: ${spring.application.name}:${vcap.application.instance_id:${spring.application.instance_id:${random.value}}}
#prefer-ip-address: true
retry:
initial-interval: 10
multiplier: 1
max-interval: 30000
max-attempts: 15

my pom.xml

`

org.springframework.boot
spring-boot-starter-parent
2.1.6.RELEASE

com.example
demo
0.0.1-SNAPSHOT
demo
Demo project for Spring Boot

<properties>
    <java.version>1.8</java.version>
    <spring-cloud.version>Greenwich.SR1</spring-cloud.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-gateway</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-netflix-ribbon</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-consul-discovery</artifactId>
    </dependency>
    <dependency>
        <groupId>org.projectlombok</groupId>
        <artifactId>lombok</artifactId>
        <optional>true</optional>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-dependencies</artifactId>
            <version>${spring-cloud.version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<build>
    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
        </plugin>
    </plugins>
</build>

`

The text was updated successfully, but these errors were encountered:

spencergibb · 2019-07-03T13:30:52Z

Please try with Greenwich.SR2

kinsey-jian · 2019-07-03T13:48:57Z

@spencergibb
Now I have changed the version to Greenwich.SR2，but it is still the same error。

my pom.xml

Dependencies

spencergibb · 2019-07-03T14:04:13Z

Please, don't use screenshots.

How do we recreate the problem?

kinsey-jian · 2019-07-04T01:53:54Z

@spencergibb I'm so sorry

this is error

2019-07-04 09:51:02.955 ERROR 1276 --- [ctor-http-nio-2] a.w.r.e.AbstractErrorWebExceptionHandler : [ee3c8217] 500 Server Error for HTTP GET "/web/provider/test"

reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response

2019-07-04 09:51:02.955 ERROR 1276 --- [tor-http-nio-11] a.w.r.e.AbstractErrorWebExceptionHandler : [64f0c49b] 500 Server Error for HTTP GET "/web/provider/test"

reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response

2019-07-04 09:51:07.772 WARN 1276 --- [tor-http-nio-10] r.netty.http.client.HttpClientConnect : [id: 0xa7c1cd87, L:0.0.0.0/0.0.0.0:52844 ! R:/172.16.61.149:9010] The connection observed an error

reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response

this is application.yml

spring:
application:
name: gateway-service
cloud:
gateway:
default-filters:
- StripPrefix=1
routes:
- id: provider_api_route
uri: lb://provider-service
predicates:
- Path=/web/**
discovery:
locator:
enabled: true
consul:
enabled: true
discovery:
instanceId: ${spring.application.name}:${vcap.application.instance_id:${spring.application.instance_id:${random.value}}}
#prefer-ip-address: true
retry:
initial-interval: 10
multiplier: 1
max-interval: 30000
max-attempts: 15

this is my pom.xml
`

org.springframework.boot
spring-boot-starter-parent
2.1.6.RELEASE

com.example
demo
0.0.1-SNAPSHOT
demo
Demo project for Spring Boot

<properties>
    <java.version>1.8</java.version>
    <spring-cloud.version>Greenwich.SR2</spring-cloud.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-gateway</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-netflix-ribbon</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-consul-discovery</artifactId>
    </dependency>
    <dependency>
        <groupId>org.projectlombok</groupId>
        <artifactId>lombok</artifactId>
        <optional>true</optional>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-dependencies</artifactId>
            <version>${spring-cloud.version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<build>
    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
        </plugin>
    </plugins>
</build>

`

spencergibb · 2019-07-04T03:50:15Z

How do we recreate the problem?

We need step by step instructions and maybe a project that recreates the problem (not pasted code).

kinsey-jian · 2019-07-04T06:04:47Z

@spencergibb ok

this is gateway project https://github.com/kinsey-jian/gateway.git

this is projecthttps://github.com/kinsey-jian/cloud-k8s.git provider

you should install consul
Launch provider-service and gateway, respectively

In the gateway projiect you can use wrk -t4 -c1024 -d6s -T5 --script=./wrk.lua --latency http://localhost:8080/web/provider/test command

Ziemowit · 2019-09-25T07:05:58Z

Hey @kinsey-jian. Did not check very deeply the description.
I just remember that we had very similar problem with "Connection prematurely closed BEFORE response".

It was caused by Apache through which traffic was flowing. Apache have reached the limits and was unable to handle next connection.
But have no idea if your gateway redirects to services which are hidden behind the Apache.

spiritme1984 · 2019-11-20T02:11:01Z

Any update? I also have this issue with my project.

Ziemowit · 2019-11-20T06:38:48Z

Any update? I also have this issue with my project.

Since 0.9.0.RELEASE of reactor netty there is a possibility
to configure the pooled connection maxIdleTime.

Netty ConnectionProvider.class:

@param maxIdleTime the {@link Duration} after which the channel will be closed (resolution: ms), if {@code NULL} there is no max idle time

Without it as I understand connections on Netty site were not closed but kept with no limits.
In our case the connection is kept with no limits whereas after 2 seconds Apache on server site closes the connection if not used. It leads to

Connection prematurely closed BEFORE response

when after 2 seconds not used connection is trying to be used again by Gateway.
At least it is my assumption after our investigation.

I have added commit which allows to define such maxIdleTime here #1411 but it is "waiting for triage".

spiritme1984 · 2019-11-25T09:18:48Z

Any update? I also have this issue with my project.

Since 0.9.0.RELEASE of reactor netty there is a possibility
to configure the pooled connection maxIdleTime.

Netty ConnectionProvider.class:

@param maxIdleTime the {@link Duration} after which the channel will be closed (resolution: ms), if {@code NULL} there is no max idle time

Without it as I understand connections on Netty site were not closed but kept with no limits.
In our case the connection is kept with no limits whereas after 2 seconds Apache on server site closes the connection if not used. It leads to

Connection prematurely closed BEFORE response

when after 2 seconds not used connection is trying to be used again by Gateway.
At least it is my assumption after our investigation.

I have added commit which allows to define such maxIdleTime here #1411 but it is "waiting for triage".

Thanks @Ziemowit.
As per your reply, I set my tomcat server never close the connection like this:
<Connector port="8080" URIEncoding="utf-8" protocol="org.apache.coyote.http11.Http11NioProtocol" maxThreads="200" connectionTimeout="-1" redirectPort="8443" />
But this issue doesn't go away... That makes me very confused. Did I miss something?

Ziemowit · 2019-11-25T09:39:18Z

I saw for sure that maxIdleTime is missing for spring-gateway and it may cause the problems.

Also do not know if it is your case but AFAIK on Apache level (so before Tomcat) there are also configuration which defines how long Apache keeps the connection open.

It is our case because for us all traffic goes additionally via Apache.
If it does not help then sorry. Had hope that manage to help.

Here we are waiting for approve our commit and ship it with HOXTON release. If it does not help we will be searching further as it impacts us too :/

spencergibb · 2019-11-26T00:59:05Z

Tomcat properties have no impact since gateway uses Netty

ifrozenice · 2019-11-27T02:09:32Z

@spencergibb
It seems to be a reactor netty problem. Please check the reactor-netty issue:Connection Closed Prematurely #413 .
how to avoid this problem in gateway?

JianJang2017 · 2019-12-17T13:44:31Z

@ifrozenice do you have some solution about this issue?

sgf1205 · 2019-12-23T10:55:04Z

reactor netty problem. Please check the reactor-ne
you can try it:

	@Bean
	public HttpClient getHttpClient(){
		ConnectionProvider connectionProvider= ConnectionProvider.elastic("gw connection pool(60s time out)", Duration.ofSeconds(60l));
		return HttpClient.create(connectionProvider);
	}

i argreen with @Ziemowit , but spring gateway current GA (2.2.1) can't config maxIdleTime in yml, i find the next verion can do it .so you can config your maxIdleTime in code as above.

spencergibb · 2020-01-14T19:33:50Z

Closed via 069f24d

creaton60 · 2020-03-30T15:15:33Z

@spencergibb

Hello, I'm still having this problem.

This is my release version

SpringCloud : Hoxton.SR3
reactor-netty : 0.9.5 RELEASE
reactor-core : 3.3.3 RELEASE

This is my routeLocator

@Bean
	public RouteLocator customRouteLocator(RouteLocatorBuilder builder, AuthFilter authFilter, AccessFilter accessFilter, ActuatorFilter actuatorFilter) {
		return builder.routes()
				.route("comments.api", p -> p
						.host(host.get("api"))
						.and().path("/comments/**")
						.filters(f -> f.rewritePath("^/comments/(?<path>.+)", "/comments/${path}")
								.filter(authFilter))
						.uri(uri.get("comment")))
				.route("op.api", p -> p
						.host(host.get("api"))
						.and().path("/op/**")
						.filters(f -> f.rewritePath("^/op/(?<path>.+)", "/op/${path}")
								.filter(authFilter))
						.uri(uri.get("op")))
				.route("campaign.api", p -> p
						.host(host.get("api"))
						.and().path("/campaigns/**")
						.filters(f -> f.rewritePath("^/campaigns/(?<path>.+)", "/campaigns/${path}")
								.filter(authFilter))
						.uri(uri.get("campaign")))
				.route("trevi.apply.api", p -> p
						.host(host.get("api"))
						.and().path("/ad/trevi/apply", "/ad/trevi/notify/installed")
						.filters(f -> f.rewritePath("^/ad/trevi/(?<path>.+)", "/${path}")
						.hystrix(c -> c.setName("trevi-apply-cmd").setFallbackUri("forward:/fallback/trevi")))
						.uri(uri.get("trevi.apply")))
				.route("trevi.track.api", p -> p
						.host(host.get("api"))
						.and().path("/ad/trevi/track")
						.filters(f -> f.rewritePath("^/ad/trevi/(?<path>.+)", "/${path}")
						.hystrix(c -> c.setName("trevi-track-cmd").setFallbackUri("forward:/fallback/trevi")))
						.uri(uri.get("trevi.track")))
				.route("trevi.imp.api", p -> p
						.host(host.get("api"))
						.and().path("/ad/trevi/reward/**")
						.filters(f -> f.rewritePath("^/ad/trevi/reward/(?<path>.+)", "/reward/${path}")
						.hystrix(c -> c.setName("trevi-imp-cmd").setFallbackUri("forward:/fallback/trevi")))
						.uri(uri.get("trevi.imp")))
				.route("iapi.api", p -> p
						.host(host.get("api"))
						.and().path("/ad/coupon-products/**")
						.filters(f -> f.addRequestHeader("Service", "trevi")
								.rewritePath("^/ad/coupon-products(?<segment>.*)", "/vp/coupon-products${segment}")
						.filter(accessFilter)
						.hystrix(c -> c.setName("coupon-api-cmd").setFallbackUri("forward:/fallback/coupon")))
						.uri(uri.get("legacy-iapi")))
				.route("iapi.dev.api", p -> p
						.host(host.get("api"))
						.and().path("/dev/ad/coupon-products/**")
						.filters(f -> f.addRequestHeader("Service", "trevi")
								.rewritePath("^/dev/ad/coupon-products(?<segment>.*)", "/vp/coupon-products${segment}")
								.filter(accessFilter))
						.uri(uri.get("legacy-dev-iapi")))
				.route("api", p -> p
						.order(999)
						.host(host.get("api"))
						.filters(f -> f.addRequestHeader("Service", "legacy-api")
						.hystrix(c -> c.setName("legacy-cmd").setFallbackUri("forward:/fallback/legacy")))
						.uri(uri.get("legacy-api")))
				.build();
	}

This is my application.yaml

spring:
  profiles: sandbox
  application:
    name: webtoon-gateway-sandbox
  cloud:
    gateway:
      globalcors:
        cors-configurations:
          '[/**]':
            allowed-origins:
              - "*"
            allowed-methods:
              - "*"
            allowed-headers:
              - "*"
            allow-credentials: true
      httpclient:
        pool:
          type: fixed
          max-connections: 500
          acquire-timeout: 75000
  boot:
    admin:
      client:
        url: http://webtoon-cloud-admin.dev.daum.net
        instance:
          prefer-ip: true

hystrix:
  threadpool:
    default:
      coreSize: 128
  command:
    default:
      execution:
        isolation:
          thread:
            timeoutInMillseconds: 5000
      circuitBreaker:
        requestVolumeThreshold: 20
        errorThresholdPercentage: 50
        sleepWindowInMilliseconds: 5000
        enabled: true

Is there anything I missed?

spencergibb · 2020-03-30T15:23:45Z

Please open a new issue and provide the stack trace

Ziemowit · 2020-04-01T05:58:55Z

@creaton60 have you tried to set the maxIdleTime property?

spring:
    gateway:
      httpclient:
        pool:
          max-idle-time: <PUT_YOUR_VALUE_HERE>ms

AFAIR by default, it is NULL so your connection will be waiting forever without closing it on the gateway and it may be closed on server site.

creaton60 · 2020-04-01T06:02:41Z

@Ziemowit
Thanks I'll try it.

iceqing · 2020-06-21T13:00:47Z

spring.cloud.gateway.httpclient.pool.max-idle-time: Time in millis after which the channel will be closed. If NULL, there is no max.

server.connection-timeout or connectionTimeout: The number of milliseconds this Connector will wait, after accepting a connection, for the request URI line to be presented. Use a value of -1 to indicate no (i.e. infinite) timeout. The default value is 60000 (i.e. 60 seconds) but note that the standard server.xml that ships with Tomcat sets this to 20000 (i.e. 20 seconds). Unless disableUploadTimeout is set to false, this timeout will also be used when reading the request body (if any).
keepAliveTimeout: The number of milliseconds this Connector will wait for another HTTP request before closing the connection. The default value is to use the value that has been set for the connectionTimeout attribute. Use a value of -1 to indicate no (i.e. infinite) timeout.

@Ziemowit Thank you very mutch. It solved my problem.
The Spring Cloud Gateway project spring.cloud.gateway.httpclient.pool.max-idle-time defalut value was null (there is no max).
my proxy target server was tomcat, server.connection-timeout default was 20000ms.
My server was set to 100ms, 100ms was to short, it cause many PrematureCloseException.

My solution:

server.connection-timeout=5000
spring.cloud.gateway.httpclient.pool.max-idle-time=2000ms

Notice: spring.cloud.gateway.httpclient.pool.max-idle-time must be less than or server.connection-timeout

reactor/reactor-netty#1092

iceqing · 2020-06-21T13:44:14Z

@spencergibb Can we set a default value?
#1787

This is an example project that reproduce the problem of PrematureCloseException:
https://github.com/iceqing/spring-cloud-gateway-example

Lovnx · 2020-09-03T08:34:14Z

https://blog.csdn.net/rickiyeat/article/details/107900585

spencergibb · 2020-09-03T12:18:49Z

Please stop posting the same thing in multiple issues

spring-projects-issues added the waiting-for-triage label Jul 3, 2019

spencergibb added waiting for feedback and removed waiting-for-triage labels Jul 3, 2019

spring-projects-issues added feedback-provided and removed waiting for feedback labels Jul 3, 2019

spencergibb added waiting for feedback and removed feedback-provided labels Jul 3, 2019

spring-projects-issues added feedback-provided and removed waiting for feedback labels Jul 4, 2019

spencergibb closed this as completed Jan 14, 2020

spencergibb removed the feedback-provided label Jan 14, 2020

violetagg mentioned this issue Jun 24, 2020

reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response reactor/reactor-netty#1092

Closed

iceqing mentioned this issue Jul 8, 2020

set defalut value #1787

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response，springboot2.1.6 #1148

reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response，springboot2.1.6 #1148

kinsey-jian commented Jul 3, 2019 •

edited

Loading

spencergibb commented Jul 3, 2019

kinsey-jian commented Jul 3, 2019

spencergibb commented Jul 3, 2019

kinsey-jian commented Jul 4, 2019 •

edited

Loading

spencergibb commented Jul 4, 2019

kinsey-jian commented Jul 4, 2019

Ziemowit commented Sep 25, 2019

spiritme1984 commented Nov 20, 2019

Ziemowit commented Nov 20, 2019 •

edited

Loading

spiritme1984 commented Nov 25, 2019

Ziemowit commented Nov 25, 2019

spencergibb commented Nov 26, 2019

ifrozenice commented Nov 27, 2019 •

edited

Loading

JianJang2017 commented Dec 17, 2019

sgf1205 commented Dec 23, 2019

spencergibb commented Jan 14, 2020

creaton60 commented Mar 30, 2020

spencergibb commented Mar 30, 2020

Ziemowit commented Apr 1, 2020

creaton60 commented Apr 1, 2020

iceqing commented Jun 21, 2020 •

edited

Loading

iceqing commented Jun 21, 2020 •

edited

Loading

Lovnx commented Sep 3, 2020

spencergibb commented Sep 3, 2020

reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response，springboot2.1.6 #1148

reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response，springboot2.1.6 #1148

Comments

kinsey-jian commented Jul 3, 2019 • edited Loading

spencergibb commented Jul 3, 2019

kinsey-jian commented Jul 3, 2019

spencergibb commented Jul 3, 2019

kinsey-jian commented Jul 4, 2019 • edited Loading

spencergibb commented Jul 4, 2019

kinsey-jian commented Jul 4, 2019

Ziemowit commented Sep 25, 2019

spiritme1984 commented Nov 20, 2019

Ziemowit commented Nov 20, 2019 • edited Loading

spiritme1984 commented Nov 25, 2019

Ziemowit commented Nov 25, 2019

spencergibb commented Nov 26, 2019

ifrozenice commented Nov 27, 2019 • edited Loading

JianJang2017 commented Dec 17, 2019

sgf1205 commented Dec 23, 2019

spencergibb commented Jan 14, 2020

creaton60 commented Mar 30, 2020

spencergibb commented Mar 30, 2020

Ziemowit commented Apr 1, 2020

creaton60 commented Apr 1, 2020

iceqing commented Jun 21, 2020 • edited Loading

iceqing commented Jun 21, 2020 • edited Loading

Lovnx commented Sep 3, 2020

spencergibb commented Sep 3, 2020

kinsey-jian commented Jul 3, 2019 •

edited

Loading

kinsey-jian commented Jul 4, 2019 •

edited

Loading

Ziemowit commented Nov 20, 2019 •

edited

Loading

ifrozenice commented Nov 27, 2019 •

edited

Loading

iceqing commented Jun 21, 2020 •

edited

Loading

iceqing commented Jun 21, 2020 •

edited

Loading