Support request compression

[cstamas]

The following snippet fails to execute with AHC 1.7.3 and 1.7.4 (using default Netty provider):

  final AsyncHttpClientConfig.Builder configBuilder = new AsyncHttpClientConfig.Builder();
  configBuilder.setCompressionEnabled( true );
  configBuilder.setRequestCompressionLevel( 6 );
  final AsyncHttpClient client = new AsyncHttpClient( configBuilder.build() );

  try
  {
    final Response response = client.prepareGet( "http://www.google.com/" ).execute().get();
    System.out.println( response.getStatusCode() + " " + response.getStatusText() );
  }
  finally
  {
    client.close();
  }

If I comment out the line configBuilder.setRequestCompressionLevel( 6 ); all is fine and snippet above works.

Also, I'd like to have some explanation what this method is meant to do. By reading sources, it is used to set "compression level" on Netty's HttpContentEncoder (wanted to raise it from default 6 to 9). How are setRequestCompressionLevel() and setCompressionEnabled() corelated?

Exception thrown by snippet above is:

java.util.concurrent.ExecutionException:
java.lang.IllegalStateException: cannot send more responses than requests
       at com.ning.http.client.providers.netty.NettyResponseFuture.abort(NettyResponseFuture.java:297)
       at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.abort(NettyAsyncHttpProvider.java:1320)
       at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.exceptionCaught(NettyAsyncHttpProvider.java:1545)
       at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:117)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:777)
       at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:143)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:777)
       at org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:143)
       at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:117)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:777)
       at org.jboss.netty.channel.SimpleChannelHandler.exceptionCaught(SimpleChannelHandler.java:163)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:137)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:777)
       at org.jboss.netty.handler.codec.replay.ReplayingDecoder.exceptionCaught(ReplayingDecoder.java:456)
       at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:117)
       at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:72)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:553)
       at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:426)
       at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:47)
       at org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:645)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:593)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:770)
       at org.jboss.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:262)
       at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:119)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:585)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:576)
       at org.jboss.netty.channel.Channels.write(Channels.java:605)
       at org.jboss.netty.channel.Channels.write(Channels.java:572)
       at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:245)
       at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.writeRequest(NettyAsyncHttpProvider.java:431)
       at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:82)
       at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:397)
       at org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:388)
       at org.jboss.netty.channel.DefaultChannelFuture.setSuccess(DefaultChannelFuture.java:332)
       at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:770)
       at org.jboss.netty.channel.socket.nio.NioWorker.processRegisterTaskQueue(NioWorker.java:250)
       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:192)
       at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
       at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
       at java.lang.Thread.run(Thread.java:680)
Caused by: java.lang.IllegalStateException: cannot send more responses than requests
       at org.jboss.netty.handler.codec.http.HttpContentEncoder.writeRequested(HttpContentEncoder.java:104)
       at org.jboss.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:262)
       at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:119)
       at org.jboss.netty.channel.Channels.write(Channels.java:605)
       at org.jboss.netty.channel.Channels.write(Channels.java:572)
       at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:245)
       at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.writeRequest(NettyAsyncHttpProvider.java:431)
       at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:82)
       at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:397)
       at org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:388)
       at org.jboss.netty.channel.DefaultChannelFuture.setSuccess(DefaultChannelFuture.java:332)
       at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:770)
       at org.jboss.netty.channel.socket.nio.NioWorker.processRegisterTaskQueue(NioWorker.java:250)
       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:192)
       ... 3 more

[slandelle]

The error message is actually wrong, Netty 3 doesn't properly handle BAD_REQUEST status code: netty/netty#441

@cstamas I know this is an old issue, but would you have a simple test case (server side) so I can investigate, please?

[slandelle]

I have a Gatling user who kindly provided a test case: https://docs.google.com/file/d/0B1zTVlhQrNiyZ01CZDFDWm5QYlk/edit

Installation instructions:

• download node: http://nodejs.org/download
• go into unzipped project root and `npm install``
• node index.js

Send POST requests to localhost:3000/post/test, XML bodies, compressed on not, will be printed in the console.

RequestBuilder requestBuilder = new RequestBuilder("POST");
requestBuilder.setUrl("http://localhost:3001/post/test");
requestBuilder.setBody("<foo>bar</foo>".getBytes());
Request request = requestBuilder.build();

AsyncHttpClientConfig config = new AsyncHttpClientConfig.Builder().setRequestCompressionLevel(6).build();
AsyncHttpClient client = new AsyncHttpClient(config);
// AsyncHttpClient client = new AsyncHttpClient(); // works perfectly when no request compression

try {
    client.executeRequest(request).get();
} finally {
    client.close();
}

Note that IllegalStateException: cannot send more responses than requests is actually Netty not properly handling BAD_REQUEST status code.

[slandelle]

NettyAsyncHttpProvider wrongly uses a HttpContentCompressor, which is a HttpContentEncoder, that, as doc states, is only used for encoding responses, not requests!

Actually, Netty doesn't support request compression yet. See netty/netty#2131 and netty/netty#2132

[slandelle]

@jfarcand As this is a Netty feature request, I don't know when it will be implemented and I doubt it will be backported on Netty 3.

I say that we remove the feature for now, both in AHC1 and AHC2. This won't cause regression is AHC1 because no user could possibly use it: it just makes the request break.

[slandelle]

Change title

[jfarcand]

Agree. Closing as won't fix.

[slandelle]

@jfarcand I disagree. I've been asked for this feature on Gatling and had to implement it on my own, and it's not great and I'd rather have it supported in Netty.

Let's just turn what first was an issue into a feature request (I did one on Netty).