Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hystrix circuit short-circuited and is OPEN #749

Closed
WeipingGuo opened this issue Apr 9, 2015 · 30 comments
Closed

Hystrix circuit short-circuited and is OPEN #749

WeipingGuo opened this issue Apr 9, 2015 · 30 comments

Comments

@WeipingGuo
Copy link

Hi,

I ran into this Exception, not sure what is the cause. The setup is fairly simple and I am doing functional testing only. All Hystrix config is defaulted with one exception: HystrixCommandProperties.Setter().withFallbackEnabled(false);

Any help and suggestions would be helpful.

Caused by: com.netflix.hystrix.exception.HystrixRuntimeException: CreateUserCommand short-circuited and failed retrieving fallback.
        at com.netflix.hystrix.AbstractCommand$20.call(AbstractCommand.java:830)
        at com.netflix.hystrix.AbstractCommand$20.call(AbstractCommand.java:798)
        at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$1.onError(OperatorOnErrorResumeNextViaFunction.java:76)
        at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:70)
        at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:70)
        at rx.Observable$ThrowObservable$1.call(Observable.java:10493)
        at rx.Observable$ThrowObservable$1.call(Observable.java:10483)
        at rx.Observable.unsafeSubscribe(Observable.java:8591)
        at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$1.onError(OperatorOnErrorResumeNextViaFunction.java:77)
        at rx.internal.operators.OperatorMap$1.onError(OperatorMap.java:48)
        at com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:318)
        at com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:310)

Thanks.
Weiping

@mattrjacobs
Copy link
Contributor

The state machine describing how Hystrix responds to requests is here: https://github.com/Netflix/Hystrix/wiki/How-it-Works#flow-chart

In your case, there have been enough errors prior to the request above that the Hystrix circuit-breaker is OPEN. When the circuit-breaker is OPEN, the request is automatically short-circuited to shed load. If a fallback is defined and enabled that logic is triggered. Since you have disabled fallback, the exception is generated and thrown to the caller.

Does that make sense, or is there anything in my explanation or the docs that could use clarification?

@WeipingGuo
Copy link
Author

Thank you very much. That helps a lot.

@learnwithjunie
Copy link

Hi @WeipingGuo @mattrjacobs What are the steps to solve this error "Hystrix circuit short-circuited and is OPEN" ?

@pcj
Copy link

pcj commented Feb 20, 2018

Seems like JIRA died today on this:

HTTP Status 500 - com.google.common.util.concurrent.UncheckedExecutionException: com.netflix.hystrix.exception.HystrixRuntimeException: TenantContextCommand short-circuited and fallback failed.
type Exception report

message com.google.common.util.concurrent.UncheckedExecutionException: com.netflix.hystrix.exception.HystrixRuntimeException: TenantContextCommand short-circuited and fallback failed.

description The server encountered an internal error that prevented it from fulfilling this request.

exception

com.atlassian.tenant.api.TenantContextException: com.google.common.util.concurrent.UncheckedExecutionException: com.netflix.hystrix.exception.HystrixRuntimeException: TenantContextCommand short-circuited and fallback failed.
	com.atlassian.tenant.impl.AbstractTenantContextProvider.setTenantId(AbstractTenantContextProvider.java:132)
	com.atlassian.jira.tenant.JiraTenantContextProviderImpl.setTenantId(JiraTenantContextProviderImpl.java:54)
	com.atlassian.jira.web.filters.WorkContextInitializationFilter.lambda$setTenantId$4(WorkContextInitializationFilter.java:211)
	java.util.Optional.ifPresent(Optional.java:159)
	com.atlassian.jira.web.filters.WorkContextInitializationFilter.setTenantId(WorkContextInitializationFilter.java:211)
	com.atlassian.jira.web.filters.WorkContextInitializationFilter.doFilter(WorkContextInitializationFilter.java:89)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:30)
	com.atlassian.jira.security.ratelimit.JiraRateLimitingFilter.lambda$doFilterWithRateLimiting$2(JiraRateLimitingFilter.java:91)
	com.atlassian.ratelimit.core.impl.AbstractRateLimiter.runWithRateLimiting(AbstractRateLimiter.java:60)
	com.atlassian.jira.security.ratelimit.JiraRateLimitingFilter.doFilterWithRateLimiting(JiraRateLimitingFilter.java:84)
	com.atlassian.jira.security.ratelimit.JiraRateLimitingFilter.doFilter(JiraRateLimitingFilter.java:70)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:30)
	com.atlassian.jira.util.zipkin.ZipkinTracingFilter.lambda$doFilter$1(ZipkinTracingFilter.java:54)
	com.github.kristofa.brave.servlet.BraveServletFilter.doFilter(BraveServletFilter.java:112)
	com.atlassian.jira.util.zipkin.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:42)
	com.atlassian.jira.web.filters.SecurityHttpHeaderRequestFilter.doFilter(SecurityHttpHeaderRequestFilter.java:45)
root cause

com.google.common.util.concurrent.UncheckedExecutionException: com.netflix.hystrix.exception.HystrixRuntimeException: TenantContextCommand short-circuited and fallback failed.
	com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
	com.google.common.cache.LocalCache.get(LocalCache.java:3937)
	com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
	com.atlassian.tenant.impl.AbstractTenantContextProvider.setTenantId(AbstractTenantContextProvider.java:123)
	com.atlassian.jira.tenant.JiraTenantContextProviderImpl.setTenantId(JiraTenantContextProviderImpl.java:54)
	com.atlassian.jira.web.filters.WorkContextInitializationFilter.lambda$setTenantId$4(WorkContextInitializationFilter.java:211)
	java.util.Optional.ifPresent(Optional.java:159)
	com.atlassian.jira.web.filters.WorkContextInitializationFilter.setTenantId(WorkContextInitializationFilter.java:211)
	com.atlassian.jira.web.filters.WorkContextInitializationFilter.doFilter(WorkContextInitializationFilter.java:89)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:30)
	com.atlassian.jira.security.ratelimit.JiraRateLimitingFilter.lambda$doFilterWithRateLimiting$2(JiraRateLimitingFilter.java:91)
	com.atlassian.ratelimit.core.impl.AbstractRateLimiter.runWithRateLimiting(AbstractRateLimiter.java:60)
	com.atlassian.jira.security.ratelimit.JiraRateLimitingFilter.doFilterWithRateLimiting(JiraRateLimitingFilter.java:84)
	com.atlassian.jira.security.ratelimit.JiraRateLimitingFilter.doFilter(JiraRateLimitingFilter.java:70)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:30)
	com.atlassian.jira.util.zipkin.ZipkinTracingFilter.lambda$doFilter$1(ZipkinTracingFilter.java:54)
	com.github.kristofa.brave.servlet.BraveServletFilter.doFilter(BraveServletFilter.java:112)
	com.atlassian.jira.util.zipkin.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:42)
	com.atlassian.jira.web.filters.SecurityHttpHeaderRequestFilter.doFilter(SecurityHttpHeaderRequestFilter.java:45)
root cause

com.netflix.hystrix.exception.HystrixRuntimeException: TenantContextCommand short-circuited and fallback failed.
	com.netflix.hystrix.AbstractCommand$22.call(AbstractCommand.java:826)
	com.netflix.hystrix.AbstractCommand$22.call(AbstractCommand.java:805)
	rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140)
	rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)
	rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)
	com.netflix.hystrix.AbstractCommand$DeprecatedOnFallbackHookApplication$1.onError(AbstractCommand.java:1473)
	com.netflix.hystrix.AbstractCommand$FallbackHookApplication$1.onError(AbstractCommand.java:1398)
	rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)
	rx.observers.Subscribers$5.onError(Subscribers.java:230)
	rx.internal.operators.OnSubscribeThrow.call(OnSubscribeThrow.java:44)
	rx.internal.operators.OnSubscribeThrow.call(OnSubscribeThrow.java:28)
	rx.Observable.unsafeSubscribe(Observable.java:10151)
	rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
	rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
	rx.Observable.unsafeSubscribe(Observable.java:10151)
	rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
	rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
	rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
	rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
	rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
	rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
	rx.Observable.unsafeSubscribe(Observable.java:10151)
	rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
	rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
	rx.Observable.unsafeSubscribe(Observable.java:10151)
	rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
	rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
	rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
	rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
	rx.Observable.unsafeSubscribe(Observable.java:10151)
	rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
	rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
	rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
	rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
	rx.Observable.unsafeSubscribe(Observable.java:10151)
	rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
	rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
	rx.Observable.unsafeSubscribe(Observable.java:10151)
	rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
	rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
	rx.Observable.unsafeSubscribe(Observable.java:10151)
	rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
	rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
	rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
	rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
	rx.Observable.unsafeSubscribe(Observable.java:10151)
	rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
	rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
	rx.Observable.unsafeSubscribe(Observable.java:10151)
	rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
	rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
	rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
	rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
	rx.Observable.subscribe(Observable.java:10247)
	rx.Observable.subscribe(Observable.java:10214)
	rx.internal.operators.BlockingOperatorToFuture.toFuture(BlockingOperatorToFuture.java:51)
	rx.observables.BlockingObservable.toFuture(BlockingObservable.java:411)
	com.netflix.hystrix.HystrixCommand.queue(HystrixCommand.java:378)
	com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:344)
	com.atlassian.tenant.service.TenantContextLoader.fetch(TenantContextLoader.java:42)
	com.atlassian.tenant.impl.AbstractServiceTenantContextProvider.fetchTenantContext(AbstractServiceTenantContextProvider.java:53)
	com.atlassian.tenant.impl.AbstractTenantContextProvider.lambda$setTenantId$2(AbstractTenantContextProvider.java:124)
	com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
	com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
	com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
	com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
	com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
	com.google.common.cache.LocalCache.get(LocalCache.java:3937)
	com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
	com.atlassian.tenant.impl.AbstractTenantContextProvider.setTenantId(AbstractTenantContextProvider.java:123)
	com.atlassian.jira.tenant.JiraTenantContextProviderImpl.setTenantId(JiraTenantContextProviderImpl.java:54)
	com.atlassian.jira.web.filters.WorkContextInitializationFilter.lambda$setTenantId$4(WorkContextInitializationFilter.java:211)
	java.util.Optional.ifPresent(Optional.java:159)
	com.atlassian.jira.web.filters.WorkContextInitializationFilter.setTenantId(WorkContextInitializationFilter.java:211)
	com.atlassian.jira.web.filters.WorkContextInitializationFilter.doFilter(WorkContextInitializationFilter.java:89)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:30)
	com.atlassian.jira.security.ratelimit.JiraRateLimitingFilter.lambda$doFilterWithRateLimiting$2(JiraRateLimitingFilter.java:91)
	com.atlassian.ratelimit.core.impl.AbstractRateLimiter.runWithRateLimiting(AbstractRateLimiter.java:60)
	com.atlassian.jira.security.ratelimit.JiraRateLimitingFilter.doFilterWithRateLimiting(JiraRateLimitingFilter.java:84)
	com.atlassian.jira.security.ratelimit.JiraRateLimitingFilter.doFilter(JiraRateLimitingFilter.java:70)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:30)
	com.atlassian.jira.util.zipkin.ZipkinTracingFilter.lambda$doFilter$1(ZipkinTracingFilter.java:54)
	com.github.kristofa.brave.servlet.BraveServletFilter.doFilter(BraveServletFilter.java:112)
	com.atlassian.jira.util.zipkin.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:42)
	com.atlassian.jira.web.filters.SecurityHttpHeaderRequestFilter.doFilter(SecurityHttpHeaderRequestFilter.java:45)
root cause

java.lang.RuntimeException: Hystrix circuit short-circuited and is OPEN
	com.netflix.hystrix.AbstractCommand.handleShortCircuitViaFallback(AbstractCommand.java:980)
	com.netflix.hystrix.AbstractCommand.applyHystrixSemantics(AbstractCommand.java:558)
	com.netflix.hystrix.AbstractCommand.access$200(AbstractCommand.java:60)
	com.netflix.hystrix.AbstractCommand$4.call(AbstractCommand.java:420)
	com.netflix.hystrix.AbstractCommand$4.call(AbstractCommand.java:414)
	rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:46)
	rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
	rx.Observable.unsafeSubscribe(Observable.java:10151)
	rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
	rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
	rx.Observable.unsafeSubscribe(Observable.java:10151)
	rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
	rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
	rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
	rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
	rx.Observable.unsafeSubscribe(Observable.java:10151)
	rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
	rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
	rx.Observable.unsafeSubscribe(Observable.java:10151)
	rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
	rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
	rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
	rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
	rx.Observable.subscribe(Observable.java:10247)
	rx.Observable.subscribe(Observable.java:10214)
	rx.internal.operators.BlockingOperatorToFuture.toFuture(BlockingOperatorToFuture.java:51)
	rx.observables.BlockingObservable.toFuture(BlockingObservable.java:411)
	com.netflix.hystrix.HystrixCommand.queue(HystrixCommand.java:378)
	com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:344)
	com.atlassian.tenant.service.TenantContextLoader.fetch(TenantContextLoader.java:42)
	com.atlassian.tenant.impl.AbstractServiceTenantContextProvider.fetchTenantContext(AbstractServiceTenantContextProvider.java:53)
	com.atlassian.tenant.impl.AbstractTenantContextProvider.lambda$setTenantId$2(AbstractTenantContextProvider.java:124)
	com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
	com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
	com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
	com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
	com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
	com.google.common.cache.LocalCache.get(LocalCache.java:3937)
	com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
	com.atlassian.tenant.impl.AbstractTenantContextProvider.setTenantId(AbstractTenantContextProvider.java:123)
	com.atlassian.jira.tenant.JiraTenantContextProviderImpl.setTenantId(JiraTenantContextProviderImpl.java:54)
	com.atlassian.jira.web.filters.WorkContextInitializationFilter.lambda$setTenantId$4(WorkContextInitializationFilter.java:211)
	java.util.Optional.ifPresent(Optional.java:159)
	com.atlassian.jira.web.filters.WorkContextInitializationFilter.setTenantId(WorkContextInitializationFilter.java:211)
	com.atlassian.jira.web.filters.WorkContextInitializationFilter.doFilter(WorkContextInitializationFilter.java:89)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:30)
	com.atlassian.jira.security.ratelimit.JiraRateLimitingFilter.lambda$doFilterWithRateLimiting$2(JiraRateLimitingFilter.java:91)
	com.atlassian.ratelimit.core.impl.AbstractRateLimiter.runWithRateLimiting(AbstractRateLimiter.java:60)
	com.atlassian.jira.security.ratelimit.JiraRateLimitingFilter.doFilterWithRateLimiting(JiraRateLimitingFilter.java:84)
	com.atlassian.jira.security.ratelimit.JiraRateLimitingFilter.doFilter(JiraRateLimitingFilter.java:70)
	com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:30)
	com.atlassian.jira.util.zipkin.ZipkinTracingFilter.lambda$doFilter$1(ZipkinTracingFilter.java:54)
	com.github.kristofa.brave.servlet.BraveServletFilter.doFilter(BraveServletFilter.java:112)
	com.atlassian.jira.util.zipkin.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:42)
	com.atlassian.jira.web.filters.SecurityHttpHeaderRequestFilter.doFilter(SecurityHttpHeaderRequestFilter.java:45)
note The full stack trace of the root cause is available in the Apache Tomcat/8.0.38 logs.

Apache Tomcat/8.0.38

@VaverDariush
Copy link

I'm having the same problem, no matter what parameters I use in @HystrixCommand, even though hystrix writes

ERROR c.n.hystrix.contrib.javanica.command.GenericCommand     - failed to process fallback is the method: 'fallbackAppendData'. exception: 'class com.myException' occurred in fallback was ignored and wrapped to HystrixBadRequestException.

the exception is not wrapped and a HystrixRuntimeException is thrown with

java.lang.RuntimeException: Hystrix circuit short-circuited and is OPEN

as cause.

@shashijais789
Copy link

I am getting the same issue:

Caused by: java.lang.RuntimeException: Hystrix circuit short-circuited and is OPEN

@jeroenhabets
Copy link

Seems like JIRA died today on this:

HTTP Status 500 - com.google.common.util.concurrent.UncheckedExecutionException: com.netflix.hystrix.exception.HystrixRuntimeException: TenantContextCommand short-circuited and fallback failed.

And Confluence Cloud died on it for my today...

@saicharan566
Copy link

seems confluence died again today

@jpita
Copy link

jpita commented Sep 26, 2018

@pcj @jeroenhabets jira and confluence down again :D

@kevin92dev
Copy link

Seems like JIRA died today on this:

HTTP Status 500 - com.google.common.util.concurrent.UncheckedExecutionException: com.netflix.hystrix.exception.HystrixRuntimeException: TenantContextCommand short-circuited and fallback failed.

And Confluence Cloud died on it for my today...

Same problem here.

@saicharan566
Copy link

HTTP Status 500 – Internal Server Error
Type Exception Report

Message TenantContextCommand short-circuited and fallback disabled.

Description The server encountered an unexpected condition that prevented it from fulfilling the request.

Exception

com.netflix.hystrix.exception.HystrixRuntimeException: TenantContextCommand short-circuited and fallback disabled.
com.netflix.hystrix.AbstractCommand.handleFallbackDisabledByEmittingError(AbstractCommand.java:1052)
com.netflix.hystrix.AbstractCommand.getFallbackOrThrowException(AbstractCommand.java:878)
com.netflix.hystrix.AbstractCommand.handleShortCircuitViaFallback(AbstractCommand.java:982)
com.netflix.hystrix.AbstractCommand.applyHystrixSemantics(AbstractCommand.java:557)
com.netflix.hystrix.AbstractCommand.access$200(AbstractCommand.java:60)
com.netflix.hystrix.AbstractCommand$4.call(AbstractCommand.java:419)
com.netflix.hystrix.AbstractCommand$4.call(AbstractCommand.java:413)
rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:46)
rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
rx.Observable.unsafeSubscribe(Observable.java:10151)
rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
rx.Observable.unsafeSubscribe(Observable.java:10151)
rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
rx.Observable.unsafeSubscribe(Observable.java:10151)
rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
rx.Observable.unsafeSubscribe(Observable.java:10151)
rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
rx.Observable.subscribe(Observable.java:10247)
rx.Observable.subscribe(Observable.java:10214)
rx.internal.operators.BlockingOperatorToFuture.toFuture(BlockingOperatorToFuture.java:51)
rx.observables.BlockingObservable.toFuture(BlockingObservable.java:411)
com.netflix.hystrix.HystrixCommand.queue(HystrixCommand.java:378)
com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:344)
com.atlassian.tenant.service.TenantContextLoader.fetch(TenantContextLoader.java:42)
com.atlassian.tenant.impl.AbstractServiceTenantContextProvider.fetchTenantContext(AbstractServiceTenantContextProvider.java:58)
com.atlassian.tenant.impl.AbstractTenantContextProvider.setTenantId(AbstractTenantContextProvider.java:132)
com.atlassian.jira.tenant.JiraTenantContextProviderImpl.setTenantId(JiraTenantContextProviderImpl.java:78)
com.atlassian.jira.web.filters.WorkContextInitializationFilter.setTenantId(WorkContextInitializationFilter.java:158)
com.atlassian.jira.web.filters.WorkContextInitializationFilter.doFilter(WorkContextInitializationFilter.java:72)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
com.atlassian.jira.security.ratelimit.AbstractJiraRateLimitingFilter.doFilter(AbstractJiraRateLimitingFilter.java:74)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
com.atlassian.jira.util.zipkin.ZipkinTracingFilter.lambda$doFilter$1(ZipkinTracingFilter.java:85)
com.github.kristofa.brave.servlet.BraveServletFilter.doFilter(BraveServletFilter.java:112)
com.atlassian.jira.util.zipkin.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:73)
com.atlassian.jira.web.filters.SecurityHttpHeaderRequestFilter.doFilter(SecurityHttpHeaderRequestFilter.java:41)
com.atlassian.jira.internalstatus.RequestStatusFilter.doFilterOnce(RequestStatusFilter.java:54)
com.atlassian.jira.web.filters.FilterOnceAbstractHttpFilter.doFilter(FilterOnceAbstractHttpFilter.java:25)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
Root Cause

java.lang.RuntimeException: Hystrix circuit short-circuited and is OPEN
com.netflix.hystrix.AbstractCommand.handleShortCircuitViaFallback(AbstractCommand.java:979)
com.netflix.hystrix.AbstractCommand.applyHystrixSemantics(AbstractCommand.java:557)
com.netflix.hystrix.AbstractCommand.access$200(AbstractCommand.java:60)
com.netflix.hystrix.AbstractCommand$4.call(AbstractCommand.java:419)
com.netflix.hystrix.AbstractCommand$4.call(AbstractCommand.java:413)
rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:46)
rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
rx.Observable.unsafeSubscribe(Observable.java:10151)
rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
rx.Observable.unsafeSubscribe(Observable.java:10151)
rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
rx.Observable.unsafeSubscribe(Observable.java:10151)
rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
rx.Observable.unsafeSubscribe(Observable.java:10151)
rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
rx.Observable.subscribe(Observable.java:10247)
rx.Observable.subscribe(Observable.java:10214)
rx.internal.operators.BlockingOperatorToFuture.toFuture(BlockingOperatorToFuture.java:51)
rx.observables.BlockingObservable.toFuture(BlockingObservable.java:411)
com.netflix.hystrix.HystrixCommand.queue(HystrixCommand.java:378)
com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:344)
com.atlassian.tenant.service.TenantContextLoader.fetch(TenantContextLoader.java:42)
com.atlassian.tenant.impl.AbstractServiceTenantContextProvider.fetchTenantContext(AbstractServiceTenantContextProvider.java:58)
com.atlassian.tenant.impl.AbstractTenantContextProvider.setTenantId(AbstractTenantContextProvider.java:132)
com.atlassian.jira.tenant.JiraTenantContextProviderImpl.setTenantId(JiraTenantContextProviderImpl.java:78)
com.atlassian.jira.web.filters.WorkContextInitializationFilter.setTenantId(WorkContextInitializationFilter.java:158)
com.atlassian.jira.web.filters.WorkContextInitializationFilter.doFilter(WorkContextInitializationFilter.java:72)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
com.atlassian.jira.security.ratelimit.AbstractJiraRateLimitingFilter.doFilter(AbstractJiraRateLimitingFilter.java:74)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
com.atlassian.jira.util.zipkin.ZipkinTracingFilter.lambda$doFilter$1(ZipkinTracingFilter.java:85)
com.github.kristofa.brave.servlet.BraveServletFilter.doFilter(BraveServletFilter.java:112)
com.atlassian.jira.util.zipkin.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:73)
com.atlassian.jira.web.filters.SecurityHttpHeaderRequestFilter.doFilter(SecurityHttpHeaderRequestFilter.java:41)
com.atlassian.jira.internalstatus.RequestStatusFilter.doFilterOnce(RequestStatusFilter.java:54)
com.atlassian.jira.web.filters.FilterOnceAbstractHttpFilter.doFilter(FilterOnceAbstractHttpFilter.java:25)
com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
Note The full stack trace of the root cause is available in the server logs.

Apache Tomcat/9.0.10

@avrebarra
Copy link

@pcj @jeroenhabets @jpita guess its time for a small 'break' :D

@Anruin
Copy link

Anruin commented Sep 26, 2018

Why did you kill Jira?

@jpita
Copy link

jpita commented Sep 26, 2018

@pcj @jeroenhabets @jpita guess its time for a small 'break' :D

@shrotavre just in time coffee 😄

@kevin92dev
Copy link

It's alive!

@husnulhamidiah
Copy link

@shrotavre no break for JIRA engineering team. I guest a couple seconds ago they read this issue. :D

@phonemkyaw
Copy link

@kevin92dev No it is not.

@smashmirrorcardboardface

someone please put 50p in the meter.

@Anruin
Copy link

Anruin commented Sep 26, 2018

@ph0nemyintkyaw for me Confluence tries to get up, but Jira still doesn't work.
You can subscribe for updates here: https://status.atlassian.com/

@DrudgeRajen
Copy link

Circuit breaker opens again in JIRA 😆

@bizzr3
Copy link

bizzr3 commented Sep 26, 2018

they should use a better breaker 🤔 like 100 AMP maybe

@Hobbit44
Copy link

Jira and confluence both still down for me

@kami4ka
Copy link

kami4ka commented Sep 26, 2018

Seems like a highest priority and impact!🙃

@toninofox
Copy link

Coffee anyone?

@TehreemNisa
Copy link

I am having the same issue on the confluence.

@Tuiituiii
Copy link

Back to work everyone !! All systems are operational

@Hobbit44
Copy link

Seems to be back up for me.

@nitish076
Copy link

nitish076 commented Aug 19, 2019

when i increase load on the service. i am getting this issue. i am using 'spring-cloud-starter-hystrix', what is the fix for this?

some error happened stacktrace:com.netflix.hystrix.AbstractCommand.handleShortCircuitViaFallback(AbstractCommand.java:979)
[]2019-08-19T12:31:31,200Z INFO c.g.a.o.a.f.ProxyEndPointFallback - Client fallback called for the cause : Hystrix circuit short-circuited and is OPEN
[]2019-08-19T12:31:31,201Z INFO c.g.a.o.a.f.ProxyEndPointFallback - Unknown exception occurred
java.lang.ClassCastException: java.lang.RuntimeException cannot be cast to org.springframework.web.client.HttpClientErrorException
at com.gdn.affiliate.outbound.api.fallback.ProxyEndPointFallback.doPostCall(ProxyEndPointFallback.java:81)
at sun.reflect.GeneratedMethodAccessor343.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at feign.hystrix.HystrixInvocationHandler$1.getFallback(HystrixInvocationHandler.java:123)
at com.netflix.hystrix.HystrixCommand$3.call(HystrixCommand.java:322)
at com.netflix.hystrix.HystrixCommand$3.call(HystrixCommand.java:318)
at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:46)
at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
at rx.Observable.unsafeSubscribe(Observable.java:10151)`

this is the stacktrace

@naranglakshay
Copy link

Facing the same issue...

@pratikdandavate
Copy link

Any updates on this issue? It works for less than 20 failed requests but when I increase the load circuit getting following error
com.netflix.hystrix.exception.HystrixRuntimeException: short-circuited and no fallback available

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests