New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HTTP read timeout from declarative client #3694
Comments
This is caused by using blocking code within Netty's event loop. The following commit resolves the issue in your application: graemerocher/micronaut-http@5d2f072 Key message: don't block the event loop |
@graemerocher thanks for your inputs. However I want to keep my application synchronous which is why it is not non-blocking in my example. What you have essentially done is made it non-blocking completely. As per my understanding (https://docs.micronaut.io/1.3.4/guide/index.html#reactiveServer) the controller in my application should be executed on IO thread which removes the possibility of blocking the event loop. Can you point me in the correct direction if I am missing something here? |
you can do that in controllers, but filters which return a publisher you need to run the operation in a different thread if you block |
Alternatively you can switch to a server that is more designed for blocking. Such as Jetty/Tomcat/etc. see https://github.com/micronaut-projects/micronaut-servlet |
Sure, I get you now. Many thanks for your help. Sorry to raise this as a bug without realising the mistake. |
I cannot use HttpClientFilter in Micronaut with a blocking Controller even if I do not block the filter. Are you saying then only solution is to switch to a blocking server as suggested above? |
What I am saying is inside your public Publisher<? extends HttpResponse<?>> doFilter(MutableHttpRequest<?> request, ClientFilterChain chain) {
return chain.proceed(request.bearerAuth(tokenIssuer.retrieveToken().getAccessToken()));
} Where you are blocking on public Publisher<? extends HttpResponse<?>> doFilter(MutableHttpRequest<?> request, ClientFilterChain chain) {
return Flowable.fromCallable(() -> tokenIssuer.retrieveToken() )
.subscribeOn(Schedulers.io())
.map(res -> chain.proceed(request.bearerAuth(res.getAccessToken())));
} This is because filters need to be able to work in either scenario (blocking or non-blocking) since they can match a controller that blocks or a controller that doesn't. If you or your team are uncomfortable with this and don't want to think about blocking vs non-blocking operations my recommendation is you consider a server that is designed around the thread per request model and not the event loop model like Micronaut's support for Jetty, Tomcat or Undertow. |
@graemerocher thanks for your swift responses, I have tried both ways as you suggested but the issue still seem to persist. See my branches here: non-blocking-httpfilter non-blocking-auth-client We do not mind non-blocking code for http client filter and respective http clients as long as it works for us. |
@ashish-sharma09 I was curious about what is wrong because we have a lot of blocking calls. There is another problem in your example: The solution is to add |
thks @dstepanov it makes sense. I expected controller to not run on the event loop thread as it is a blocking controller, but it does so because of AuthValidator as you pointed out. The issue seems to go away with non-blocking auth client and filter and offloading AuthValidator (and Controller as a result) processing on a separate thread pool using Many thanks again for pointing out important points and mistakes here @graemerocher and @dstepanov 👍 As I see in the documentation of latest Micronaut version (we use 1.3.4) though that the design has changed around it and even blocking controller runs on event loop thread. However one can choose to run on separate pool using ExecuteOn. Am I right in saying that there has been major shift in the design between 1.3.x and 2.x.x version of Micronaut around this? |
The issue seems to be with Micronaut declarative http clients where they start erroring out with io.micronaut.http.client.exceptions.ReadTimeoutException: Read Timeout when several concurrent requests are in play (more than 20 seems to reproduce issue consistently) using the clients.
This seem to only happen (wierdly) when a declarative client is used in conjunction with Micronaut security, Micronaut HttpClientFilter and Controller.
Initially we thought that the issue is exactly as reported here: #2905 but trying out the workaround suggested in this post does not fix the issue for us.
Task List
Steps to Reproduce
Please see and run StatusIntegationSpec test at: https://github.com/ashish-sharma09/micronaut-http to reproduce the issue
./gradlew test -debug
to see stacktrace (provided below) being produced.Stacktrace:
io.micronaut.http.client.exceptions.ReadTimeoutException: Read Timeout at io.micronaut.http.client.exceptions.ReadTimeoutException.<clinit>(ReadTimeoutException.java:26) at io.micronaut.http.client.DefaultHttpClient.lambda$null$29(DefaultHttpClient.java:1090) at io.reactivex.internal.operators.flowable.FlowableOnErrorNext$OnErrorNextSubscriber.onError(FlowableOnErrorNext.java:103) at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onError(RxInstrumentedSubscriber.java:83) at io.reactivex.internal.operators.flowable.FlowableTimeoutTimed$TimeoutSubscriber.onTimeout(FlowableTimeoutTimed.java:139) at io.reactivex.internal.operators.flowable.FlowableTimeoutTimed$TimeoutTask.run(FlowableTimeoutTimed.java:170) at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66) at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Expected Behaviour
Referring to the repo provided:
Test should pass with no error being thrown ie all responses from Status endpoint should return 200.
Actual Behaviour
Referring to the repo provided:
When status endpoint is hit with 20 concurrent requests, the http client fails with ReadTimeout error.
Environment Information
Example Application
The text was updated successfully, but these errors were encountered: