Curl hangs when Header "Expect:100-continue" is set on POST > 1024 bytes

[runabol]

I'm using Spring Cloud Gateway Finchley.M9.

I noticed that when I proxy a large POST the gateway times out:

$ curl -X POST -H "Content-Type:application/json" -d '[{ "id": "0001", "type": "donut", "name": "Cake", "ppu": 0.55, "batters": { "batter": [{ "id": "1001", "type": "Regular" }, { "id": "1002", "type": "Chocolate" }, { "id": "1003", "type": "Blueberry" }, { "id": "1004", "type": "Devils Food" } ] }, "topping": [{ "id": "5001", "type": "None" }, { "id": "5002", "type": "Glazed" }, { "id": "5005", "type": "Sugar" }, { "id": "5007", "type": "Powdered Sugar" }, { "id": "5006", "type": "Chocolate with Sprinkles" }, { "id": "5003", "type": "Chocolate" }, { "id": "5004", "type": "Maple" } ] }, { "id": "0002", "type": "donut", "name": "Raised", "ppu": 0.55, "batters": { "batter": [{ "id": "1001", "type": "Regular" }] }, "topping": [{ "id": "5001", "type": "None" }, { "id": "5002", "type": "Glazed" }, { "id": "5005", "type": "Sugar" }, { "id": "5003", "type": "Chocolate" }, { "id": "5004", "type": "Maple" } ] }, { "id": "0003", "type": "donut", "name": "Old Fashioned", "ppu": 0.55, "batters": { "batter": [{ "id": "1001", "type": "Regular" }, { "id": "1002", "type": "Chocolate" } ] }, "topping": [{ "id": "5001", "type": "None" }, { "id": "5002", "type": "Glazed" }, { "id": "5003", "type": "Chocolate" }, { "id": "5004", "type": "Maple" } ] } ]' https://my.service.name.behind.the.gateway/my/path

curl: (52) Empty reply from server

This does not happen with smaller body payloads:

$ curl -X POST -H "Content-Type:application/json" -d '[{ "id": "0001", "type": "donut", "name": "Cake", "ppu": 0.55 }, { "id": "0002", "type": "donut", "name": "Raised", "ppu": 0.55, "batters": { "batter": [{ "id": "1001", "type": "Regular" }] }, "topping": [{ "id": "5001", "type": "None" }, { "id": "5002", "type": "Glazed" }, { "id": "5005", "type": "Sugar" }, { "id": "5003", "type": "Chocolate" }, { "id": "5004", "type": "Maple" } ] }, { "id": "0003", "type": "donut", "name": "Old Fashioned", "ppu": 0.55 } ]' https://my.service.name.behind.the.gateway/my/path

{
  "timestamp": "2018-04-10T20:46:45Z",
  "status": 404,
  "error": "Not Found",
  "message": "No message available",
  "path": "/my/path"
}

[spencergibb]

What version?

Can you tell me how long?

Are you using hystrix?

[runabol]

Finchley.M9

real    0m59.307s
user    0m0.017s
sys     0m0.014s

Not using Hystrix.

[spencergibb]

Works fine if I use httpie.org

http POST :8080/post Host:www.myhost.org @test.json 

These are the headers set by httpie

Accept: application/json, */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 1518
Content-Type: application/json
Host: www.myhost.org
User-Agent: HTTPie/0.9.8

something to do with curl and reactor and/or netty? @smaldini

[runabol]

I think that I found the culprit.

Add the Expect:100-continue header to your httpie request. curl adds that header automatically on a payload that is greater than 1024 bytes.

Ref: https://gms.tf/when-curl-sends-100-continue.html

[runabol]

Just tried it again today with Finchley.RELEASE and I wasn't able to reproduce the issue.

[spencergibb]

thanks for the feedback @creactiviti!

[dan-nawrocki]

This appears to be back for me. I think the regression is caused by 2816605. I believe it disappeared in Finchley.RELEASE because there is no Content-Type for a HTTP 100 response, triggering a NullPointerException that somehow caused both the HTTP 100 response and the target HTTP response to be returned back to curl (?).

I don't know much about spring reactive, but my naive guess from debugging is that spring-cloud-gateway is responding to curl with the HTTP 100 response, and not forwarding the actual response from the target. Does spring-cloud-gateway assume that each request will receive a single response from the target?

I can get curl to work by disabling the 100-Continue feature with -H 'Expect:'.

Edit: @spencergibb any ideas?

[runabol]

I was able to reproduce and find a workaround to the issue mentioned by @dan-nawrocki.

It seems like the issue was resolved by the netty team: reactor/reactor-netty#293 but for some reason the Expect header still gets forwarded to the target.

The target as a result returns a 100 Continue (as it should) which results in a NullPointerException on NettyRoutingFilter:117 as it's missing a Content-Type on the response:

exchange.getAttributes().put("original_response_content_type", headers.getContentType());

As a workaround I was able to prevent Spring Cloud Gateway from forwarding the Expect header by adding the following to my route:

.filters((t)->t.removeRequestHeader("Expect"))

[spencergibb]

The NPE has been fixed in snapshots

[Odyseja]

Hi, in 2.0.2.BUILD-SNAPSHOT problem still occurs.
There is no NPE now, but curl still hangs when request body is too big.

[spencergibb]

@Odyseja what version of spring boot and reactor-netty are you using? The fix is in reactor-netty 0.7.5

[Odyseja]

Spring boot's version is 2.0.3.RELEASE and reactor-netty's version is 0.7.8.RELEASE

[jinchaoxiang]

me too

[Allen57]

Same here with Spring boot 2.0.6.RELEASE and reactor-netty 0.7.10.RELEASE.
Adding a filter to remove the 'Expect' header (as suggested by @creactiviti) solved the issue for me.