Gateway should handle header 'Expect 100-continue' compliant with rfc2616

[michaldo]

• If a proxy receives a request that includes an Expect request-
  header field with the "100-continue" expectation, and the proxy
  either knows that the next-hop server complies with HTTP/1.1 or
  higher, or does not know the HTTP version of the next-hop
  server, it MUST forward the request, including the Expect header
  field.

Steps to reproduce problem

Target server: Spring Boot 2.1.8 (Tomcat)

@SpringBootApplication @RestController 
public class MyDownstreamApp {
   public static void main(String[] args) {
      SpringApplication.run(MyDownstreamApp.class, args);
  }

  @PutMapping("/put")
  String put(@RequestParam String x) {
      System.out.println("x: " +x);
      return "hello";
  }
}

Cloud gateway: Greenwich SR3

@SpringBootApplication
public class MyGatewayApp {

    @Bean
    RouteLocator myRoutes(RouteLocatorBuilder builder) {
        return builder.routes()
                .route(p -> p
                        .path("/put")
                        .uri("http://localhost:8080"))
                .build();
    }

    public static void main(String[] args) {
        SpringApplication.run(MyGatewayApp.class, "--server.port=8082");
    }
}

When target server is called directly:
curl localhost:8080/put -X PUT -H 'Expect: 100-continue' -d x=foo1 -v

communication flow is OK

1. Client sends Expect: 100-continue
2. Server reponds 100-continue
3. Client sends complete request
4. Server responds hello

> Expect: 100-continue
> Content-Length: 6
> Content-Type: application/x-www-form-urlencoded
>
< HTTP/1.1 100
* We are completely uploaded and fine
< HTTP/1.1 200
< Content-Type: text/plain;charset=UTF-8
< Content-Length: 5

When target server is called through gateway:
curl localhost:8082/put -X PUT -H 'Expect: 100-continue' -d x=foo1 -v

communication flow is not OK. My wireshark investigation

1. Client sends Expect: 100-continue to gateway
2. Gateway responds 100-continue
3. Client sends complete request to gateway
4. I'm not sure 100% what happens in point 4
5. Target server responds 100-continue to gateway
6. Gateway sends complete request to target server
7. Target server responds hello
8. Gateway forwards 100-continue to client
9. Client waits for response hello but it is not forwarded

> Expect: 100-continue
> Content-Length: 6
> Content-Type: application/x-www-form-urlencoded
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 100 Continue
(hang up)

Response 100 Continue is received 2 times - once generated by gateway itself, second forwarded from target server, but true response never reach client.

Worth to notice that workaround is filter our header 'Expect 100-continue` when gateway forwards request to target server

[stalelabs]

This is seriously very weird behaviour. We have spring cloud gateway in production environment and for clients which send Expect: 100-continue are not getting proper response.
I don't know how is this possible but during peak times the response for the route is also wrong.

For ex : Two routes A and B and they both call expecting 100. Route A gets 100 status code and the the payload response is sent to route B.

[spencergibb]

reactor/reactor-netty#722

[spencergibb]

I believe this is blocked by the reactor netty issue listed above.

[violetagg]

@spencergibb Reactor Netty will respond automatically with 100 Continue ONLY when the IO handler asks for the body but there was no response with 100 Continue before that.

https://github.com/reactor/reactor-netty/blob/89ae010a84e25925952e2f82ece7fae8c015742c/src/main/java/reactor/netty/http/server/HttpServerOperations.java#L292-L301

In Spring Gateway I see the following:

spring-cloud-gateway/spring-cloud-gateway-core/src/main/java/org/springframework/cloud/gateway/filter/NettyRoutingFilter.java

Line 147 in 73fd528

return nettyOutbound.send(request.getBody().map(this::getByteBuf));

In the implementation the incoming data is requested always without checking whether there is Expect header

[spencergibb]

To document, request.getBody() is called in webflux initialization very early on before gateway can respond to any header here https://github.com/spring-projects/spring-framework/blob/master/spring-web/src/main/java/org/springframework/web/server/adapter/DefaultServerWebExchange.java#L131

So it still looks blocked but at the framework level now @rstoyanchev

[rstoyanchev]

That is not an actual call but deferred initialization in case of a call to exchange.getFormData(). To trigger that something must call getFormData().

[spencergibb]

I'm able to send a 100 Continue from gateway (or let the downstream service send it), but it seems the request is terminated and the client never sends the body. I'm not sure where in the stack the problem is.

[SAM-RANDOM001]

Is there any other work around in springboot to handle the Expect: 100-continue header.

[baev]

Is there any other work around in springboot to handle the Expect: 100-continue header.

From issue description:

Worth to notice that workaround is filter our header 'Expect 100-continue` when gateway forwards request to target server

So you can simply add route filter that removes Expect header

.filters(f -> f.removeRequestHeader("Expect"))

or

filters:
        - RemoveRequestHeader=Expect

[matejuh]

@spencergibb gateway sends 100 Continue, but also for the final response. I have Wiremock configured to return 201. In case of direct call to Wiremock:

$ curl localhost:8082/huge-request -X POST -H 'Content-Type: text/plain'  --data "@10KB.txt" -v
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8082 (#0)
> POST /huge-request HTTP/1.1
> Host: localhost:8082
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Type: text/plain
> Content-Length: 10240
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 201 Created
< Matched-Stub-Id: 9cd25804-832d-45bf-9b25-91990ac9c4c5
< Matched-Stub-Name: for testing
< Vary: Accept-Encoding, User-Agent
< Transfer-Encoding: chunked
< Server: Jetty(9.4.20.v20190813)
< 
* Connection #0 to host localhost left intact
* Closing connection 0

In case of call through gateway, two 100 Continue are returned and connection timeouts:

 $ curl localhost:8080/huge-request -X POST -H 'Content-Type: text/plain'  --data "@10KB.txt" -v
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> POST /huge-request HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Type: text/plain
> Content-Length: 10240
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 100 Continue

[paskos]

I just updated spring-boot to version 2.4.3 which depends on reactor-netty version 1.0.4 and netty version 4.1.59
The RemoveRequestHeader work around does not work any more.
I tried to create a gatewayFilter that sets response status to 100 and complete the response but I get:

Connection prematurely closed BEFORE response; nested exception is reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
org.springframework.web.reactive.function.client.WebClientRequestException: Connection prematurely closed BEFORE response; nested exception is reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
	at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:137)

[michaldo]

That is strange, because my experiment shows that spring-boot version 2.4.3 & spring cloud version 2020.0.1 fixes the problem

The problem is still visible for spring-boot version 2.3.9.RELEASE and spring cloud Hoxton.SR9

I vote for close the problem

[paskos]

I reckon I haven't given enough details about what I'm trying to do.
We have several micro-services behind a spring-cloud-gateway which is accessed by mobile games.
We want the gateway to handle Expect 100-continue headers and respond with 100.
We don't want the gateway to pass the request to downstream services in this case.
Before spring-boot 2.4.3, we added RemoveRequestHeader and the gateway worked as required.
Since upgrading to spring-boot 2.4.3 it passes the request downstream which we don't want it to do.
I tried to implement a gateway filter responsible to intercept Expect 100-continue requests, set the response to 100 and complete the response.
I can't make it work; I alwys the following exception:
Connection prematurely closed BEFORE response; nested exception is reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response org.springframework.web.reactive.function.client.WebClientRequestException: Connection prematurely closed BEFORE response; nested exception is reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:137)

[spencergibb]

@paskos I'm going to say your issue is different than the original intent of this issue. Can you post your filter?

[paskos]

ExpectContinueGatewayFilterFactory
I changed the package but the code is exactly the same.

I configured it in the gateway's default filters:

spring:
  cloud:
    gateway:
      default-filters:
        - ExpectContinue
        

I found the changes in behaviour thanks to an integration test in my gateway that uses wiremock to simulate a downtstream service.
WebTestClient sends a http request with the Expect 100-continue header and then the tests calls wiremock verify() to make sure that no requests were received by it.

When my gateway depends on spring-boot version 2.4.2 and spring-cloud 2020.0.1 my test passes.
When I upgrade to spring-boot version 2.4.3 the test fails because wiremock did receive the request.

[violetagg]

This might be related to this fix reactor/reactor-netty#1492

[paskos]

Thank you for the link.
@violetagg @spencergibb I'm far from proficient with Netty, spring-cloud-gateway and reactor or reactive programming.
Could you, please, point me in the direction on how to implement Expect 100-continue at the gateway level where the request is not sent downstream ?
Thanks a lot in advance

[belaszalontai]

Currently we are working on a project using Spring Cloud Gateway with dynamic routing and custom load balancing.

We faced the same issue and after some investigation we realized that as michalod said in this comment: #1337 (comment)

The 100-continue feature works perfectly in Spring Cloud Gateway with the latest version of related dependencies.
(for us: spring-cloud-starter-gateway v3.0.4 spring-cloud-gateway-webflux v3.0.4 spring-boot v2.5.5)

So I think this issue can be closed.

And a late response to paskos:

We want the gateway to handle Expect 100-continue headers and respond with 100.
We don't want the gateway to pass the request to downstream services in this case.

This is wrong behaviour because if the mobile game requests your gateway with Expect: 100-continue header without Body, then it keeps alive the connection and waits for a response. If the status of the response is 100 Continue, then the mobile game tries to send the rest of the request (actually the Body) in the same TCP connection then waits for a final response (for example: 200 Ok). If the gateway do not pass the request, then the mobile game will hang until timeout.

If you would like to reject the requests from mobile games which contains Expect: 100-continue header then in accordance with the RFC standards your gateway has to respond 417 Expectation Failed. If the mobile game implemented the RFC properly then it will send the full request again without Expect header.

Here is the code you can use in your custom ExpectContinueGatewayFilterFactory if you want to implement this way.

String expectHeader = exchange.getRequest().getHeaders().get("Expect").get(0);

if (expectHeader != null && "100-continue".equals(expectHeader)) {
    exchange.getResponse().setStatusCode(EXPECTATION_FAILED); // Reject with Status Code 417
    return exchange.getResponse().setComplete();
}

this code is also a workaround for Spring Cloud Gateway apps using spring-boot version before 2.4.3 if Client hangs in the 100-continue flow.

[michael-o]

I reckon I haven't given enough details about what I'm trying to do. We have several micro-services behind a spring-cloud-gateway which is accessed by mobile games. We want the gateway to handle Expect 100-continue headers and respond with 100. We don't want the gateway to pass the request to downstream services in this case. Before spring-boot 2.4.3, we added RemoveRequestHeader and the gateway worked as required. Since upgrading to spring-boot 2.4.3 it passes the request downstream which we don't want it to do. I tried to implement a gateway filter responsible to intercept Expect 100-continue requests, set the response to 100 and complete the response. I can't make it work; I alwys the following exception: Connection prematurely closed BEFORE response; nested exception is reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response org.springframework.web.reactive.function.client.WebClientRequestException: Connection prematurely closed BEFORE response; nested exception is reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:137)

This clearly violates https://datatracker.ietf.org/doc/html/rfc7231#section-5.1.1. If the proxy cannot handle the expect handle it must ask the origin. So I expect the API gateway to forward all request headers to the origin server if it cannot make a decision based on the headers.