Netty4 HTTP does not support "Expect: 100-continue" header

[tlrx]

When using Netty4 HTTP transport type I have the following issue:

Netty4

Starting master using http.type: netty4

bin/elasticsearch --E http.type=netty4

Creating a document with a medium size JSON document using cUrl (sample document is here):

curl -v -XPOST 'localhost:9200/samples/sample/0' -d '{
   "title":"My awesome book",
   "pages":456,
   "price":27.99,
   "timestamp":1428582942867,
    ..... 30 other fields...
}'

Here is the curl output:

Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 9200 (#0)
> POST /samples/sample/0 HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/7.47.0
> Accept: */*
> Content-Length: 1681
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
> 
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server

And the elasticsearch logs:

[2016-08-05 17:46:13,354][WARN ][http.netty4              ] [UqG1hcq] caught exception while handling client http traffic, closing connection [id: 0x0ca9a8ca, L:/127.0.0.1:9200 - R:/127.0.0.1:43208]
java.lang.UnsupportedOperationException: unsupported message type: DefaultFullHttpResponse (expected: ByteBuf, FileRegion)
    at io.netty.channel.nio.AbstractNioByteChannel.filterOutboundMessage(AbstractNioByteChannel.java:260)
    at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:799)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1291)
    at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748)
    at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:811)
    at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:824)
    at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:804)
    at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:841)
    at io.netty.handler.codec.MessageAggregator.decode(MessageAggregator.java:222)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:350)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:350)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:350)
    at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:350)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:571)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:474)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:428)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:398)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:877)
    at java.lang.Thread.run(Thread.java:745)

Update: as a workaround for users, forcing the Expect header to empty makes the request succeed:

curl -v -H "Expect:" -XPOST 'localhost:9200/samples/sample/0' -d '...'

Netty3

It works fine with http.type: netty3

bin/elasticsearch --E http.type=netty3

Creating the document:

curl -v -XPOST 'localhost:9200/samples/sample/0' -d '{
   "title":"My awesome book",
   "pages":456,
   "price":27.99,
   "timestamp":1428582942867,
    ..... 30 other fields...
}'

Here is the curl output:

Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 9200 (#0)
> POST /samples/sample/0 HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/7.47.0
> Accept: */*
> Content-Length: 1681
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 201 Created
< Location: /samples/sample/0
< Content-Type: application/json; charset=UTF-8
< Content-Length: 142
< 
* Connection #0 to host localhost left intact
{"_index":"samples","_type":"sample","_id":"0","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"created":true}%   

[tlrx]

For what it worth, it looks like the Netty4HttpServerTransport correctly initializes the Channel using a HttpObjectAggregator which seems to be in charge of handling the Expect: 100-continue header with the io.netty.handler.codec.http.HttpObjectAggregator.newContinueResponse() method

[tlrx]

It seems to be a Netty issue and I have a potential fix. I'm trying to create a test to reproduce this and verify my fix. If successful I'll create a pull request on Netty project.

[jasontedor]

I'm not convinced this is a Netty issue, I think the handlers are just ordered incorrectly. I opened #19904.

[tlrx]

@jasontedor I agree - that's also my conclusion after some more digging. I was about to submit the same change as you.