-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HystrixRuntimeException: TestCommand fallback execution rejected #1357
Comments
What did you define in your fallback? |
In my fallback method, I am hitting another url using REST call and returning a xml response |
The key line in the logs is:
This means that too many fallbacks were executing concurrently. Since the goal of Hystrix is to protect application threads, and fallbacks may run on the caller thread, Hystrix needs to limit the amount of concurrent fallbacks. Whenever this situation occurs, the fallback is not run, and an exception is returned to the caller. In general, the purpose of a fallback is to provide substitute work that is less costly than the original work. Here are 2 options for proceeding:
|
I did set these two properties of my TestCommand. It got fixed. This issue is due to when more number of requests go to fallback method concurrently. Hystrix has certain limit for the concurrent request. .withFallbackIsolationSemaphoreMaxConcurrentRequests(Integer.MAX_VALUE) |
By setting those properties, you're throwing away the ability to use a concurrency limit on semaphore-isolated commands and on fallbacks. If your system starts timing out, fallbacks will begin to run on your application threads, and the setting you made above (fallbackIsolation = Integer.MAX_VALUE) will mean that all application threads may get consumed by your fallbacks. I strongly advise against doing this - the whole point of using Hystrix is to set up these sort of guardrails such that your system cannot become unhealthy. As I mentioned above, when you build a fallback, you should aim to do less work in the fallback than in the normal execution path. That means that failing-fast will actually relieve load on your system. |
I agree with @mattrjacobs , fallback should be mean of alternate work at a less cost. Generally, I use a chain of fallbacks (wrapped in Hystrix commands) to make sure we have a defined fallback pattern with each having less cost of work. |
Closing due to inactivity. Please re-open if there's more to discuss. |
Hi , hystrix: Our fallback is static response there is no network call i have few questions can any one please help me Few times we saw fall back rejected due to que size , check below few lines of stack trace -> In above we didn't see any log which says that queue size is reached maximum level, but still it rejected |
I am using Hystrix 1.5.5 version. When I do load testing of bigger load like 1000 thread/second, all the requests are going through fallback method. Meanwhile, I am getting below exception too. Why do I get this below exception. TestCommand is my custom Hystrix class.
Caused by: com.netflix.hystrix.exception.HystrixRuntimeException: TestCommand fallback execution rejected.
at com.netflix.hystrix.AbstractCommand.handleFallbackRejectionByEmittingError(AbstractCommand.java:1026)
at com.netflix.hystrix.AbstractCommand.getFallbackOrThrowException(AbstractCommand.java:858)
at com.netflix.hystrix.AbstractCommand.handleThreadPoolRejectionViaFallback(AbstractCommand.java:976)
at com.netflix.hystrix.AbstractCommand.access$400(AbstractCommand.java:59)
at com.netflix.hystrix.AbstractCommand$12.call(AbstractCommand.java:593)
at com.netflix.hystrix.AbstractCommand$12.call(AbstractCommand.java:587)
at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140)
at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:72)
at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:72)
at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$3.onError(AbstractCommand.java:1173)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:54)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
The text was updated successfully, but these errors were encountered: