CircuitBreaker cannot close during constant traffic

[wojciech-piotrowiak]

I'm not sure if this is a bug, I tried find a help on stackoverflow, but without success: link

In a few words I have a module that is trying to connect other server by REST and use CircuitBreaker between. Frequency will be configured with Cron, but let assume 10 seconds. During tests I find out that if error count is equal to maxAttempts then it is not possible that CircuitBreaker will close if requests are going with the same frequency. If there is a pause between requests that is as long as resetTimeout then circuit is closing again.

I created simple Java project with test that reproduce this problem here: github
I also managed to create similar simulator with Hystrix and it runs without any problems.

[dbusley]

I am having the same issue on version 1.2.2.RELEASE

[av206a]

I am also having the same issue under the constant load. Is there any fix available for this?

[alexlmikh]

I have the same issue with 1.2.1.RELEASE.

I'm not sure if this is a bug

I'm sure that this is indeed a bug, having a "circuit breaker" which never recovers, if client requests keep on coming, pretty much defeats the purpose of having a circuit breaker in the first case.

[garyrussell]

I am not really familiar with the code, but it looks like the problem is here in the CircuitBreakerRetryContext...

				if (time > this.timeout) {
					logger.trace("Closing");
					this.context = createDelegateContext(policy, getParent());
					this.start = System.currentTimeMillis();
					retryable = this.policy.canRetry(this.context);
				}
				else if (time < this.openWindow) {
					if ((Boolean) getAttribute(CIRCUIT_OPEN) == false) {
						logger.trace("Opening circuit");
						setAttribute(CIRCUIT_OPEN, true);
					}
					this.start = System.currentTimeMillis(); // <<<<<<<<<<<<<<<<<<<
					return true;
				}

The timer starts again on each failure.

Changing it to set the timer only when the circuit first goes open...

				if (time > this.timeout) {
					logger.trace("Closing");
					this.context = createDelegateContext(policy, getParent());
					this.start = System.currentTimeMillis();
					retryable = this.policy.canRetry(this.context);
				}
				else if (time < this.openWindow) {
					if ((Boolean) getAttribute(CIRCUIT_OPEN) == false) {
						logger.trace("Opening circuit");
						setAttribute(CIRCUIT_OPEN, true);
						this.start = System.currentTimeMillis();
					}
					return true;
				}

Seems to fix it (at least the test provided by the OP).

cc/ @dsyer

[BijuSs]

Hi Guys,

Is there anyway to set the resetTimeOut from the first Circuit OPEN state, not the latest? I am also facing the same issue.

Thanks

[dsyer]

Maybe someone could send a pull request?

[BijuSs]

Hi @dsyer ,

Is this issue to be fixed in 1.2.6 version? If so, could you please let me know when the new version is supposed to be released?

Thanks

[wojciech-piotrowiak]

@BijuSs This problem is still visible in latest 1.3.0

@dsyer Will send pull request with solution provided by @garyrussell this week.
That idea works, but not sure why it is after 15 seconds, when open timeout is 10 seconds and reset timeout 20 seconds:

13329 ms after last failure
before ok
----/ ---- open, retry count:3
recovery
---------- close, retry count:3
after ok

15232 ms after last failure
before ok
----/ ---- open, retry count:3
real service called
---------- close, retry count:0
after ok

[wojciech-piotrowiak]

@dsyer @garyrussell
PR is ready: #203

[wojciech-piotrowiak]

Friendly reminder about open PR :)
@dsyer @garyrussell

[dsyer]

#203 is merged (and cherry picked to 1.2.x). You could try it in snapshots.

[BijuSs]

@wojciech-piotrowiak @dsyer
Sorry, if I have missed anything here. I know that the fix is merged to the spring-retry project.
But please let me know which release version this fix is available?

Thanks

[dsyer]

It's marked as fixed in 1.2.6 (which hasn't been released yet). It will also be in 1.3.1. You can try it in snapshots.