-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] Blob Storage: Netty HttpClient is slow when downloading via an authenticated proxy server #19415
Comments
@anuchandy @alzimmermsft Any thoughts on whether this might be the case for the netty client? |
Yes, this appears to be a difference between how our Netty and OkHttp HTTP clients wrappers handle authenticated proxying. |
Additional context on this issue, this should only happen for the first connection using the authenticated proxy while subsequent connections should eagerly apply authentication. The first CONNECT does not as it supports both basic and digest authentication and does not eagerly apply basic even though it is possible. One change here other than more quickly retrying on a 407 exception would be applying a basic authentication eagerly (though guards may be added to only do this on a secure connection). |
Thank you for looking into this. Applying the authentication eagerly for secure connections would not help in our specific use case (but might benefit others). Quickly retrying like OkHttp does would be completely sufficient for us. |
I can still reproduce this with azure-storage-blob 12.11.0-beta.2 and azure-core-http-netty 1.9.1. |
Hi @sesu-bio, do you have a reproduction of this issue available? I've added testing to ensure when a |
I am still using the above code. This is from my pom.xml:
Looks like the
Netty still takes about 5s longer for the first download:
Maybe the default retry delay is actually 5s? |
Hi @sesu-bio, took a look into this again. It appears that exception propagation changed leading to our logic to unwrap I've submitted #20675 to better resolve this issue. I did notice during testing though that the initial request through Netty does take longer than OkHttp (likely due to the additional initialization costs it has with connection pooling, etc), so some of the differences you may be seeing in the repro you gave may be due to that. |
@sesu-bio |
@alzimmermsft Using the latest dependency versions, the Netty http client has gotten a lot faster in the described situation. The remaining difference is most likely caused by the effects you mentioned in your previous comment.
A warning about the 407 HTTP status is still logged but this is only a minor annoyance. I consider the issue resolved. Thank you very much! |
Fix small logical mistake in automation rules examples (Azure#19415) * Fix small logical mistake in automation rules examples * Update readme.md Co-authored-by: Wei Dong <40835867+dw511214992@users.noreply.github.com>
Describe the bug
When downloading a file via an authenticated proxy server, the netty HttpClient is noticably slower than the OkHttp one. It also outputs a warning "Failed to connect to proxy." I suspect that it interpretes the 407 Proxy Authentication Required from the proxy server as an error and waits a while until it retries.
Exception or Stack Trace
[reactor-http-nio-1] WARN reactor.netty.http.client.HttpClientConnect - [id: 0xe9730fb2, L:/169.254.111.2:61622 - R:/169.254.111.2:3128] The connection observed an error io.netty.handler.proxy.ProxyConnectException: Failed to connect to proxy. at com.azure.core.http.netty.implementation.HttpProxyHandler.handleResponse(HttpProxyHandler.java:239) at io.netty.handler.proxy.ProxyHandler.channelRead(ProxyHandler.java:258) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) [reactor-http-nio-1] WARN reactor.netty.http.client.HttpClientConnect - [id: 0xe9730fb2, L:/169.254.111.2:61622 ! R:/169.254.111.2:3128] The connection observed an error io.netty.handler.proxy.ProxyConnectException: Failed to connect to proxy. at com.azure.core.http.netty.implementation.HttpProxyHandler.handleResponse(HttpProxyHandler.java:239) at io.netty.handler.proxy.ProxyHandler.channelRead(ProxyHandler.java:258) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)
To Reproduce
Run the below code snippet running an authenticated proxy server on port 3128.
Result for a small text file:
Code Snippet
Expected behavior
Netty HttpClient should be about as fast as OkHttp and not log a warning.
Setup (please complete the following information):
Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report
The text was updated successfully, but these errors were encountered: