[Hystrix 1.5.13] The state of a CircuitBreaker will be still OPEN permanently

[yanglifan]

The issue should be caused by pull #1621, and already mentioned by issue #1853.

I have demo this issue with a test.

static class TestCommand extends HystrixCommand<Void> {
    private static AtomicInteger count = new AtomicInteger();

    private volatile boolean hasAddCount = false;
    private CountDownLatch countDownLatch;

    TestCommand(CountDownLatch countDownLatch) {
        super(Setter.withGroupKey(HystrixCommandGroupKey.Factory.asKey("test"))
                .andThreadPoolPropertiesDefaults(HystrixThreadPoolProperties.Setter().withCoreSize(N_THREADS)));
        this.countDownLatch = countDownLatch;
    }

    @Override
    protected Void run() throws Exception {
        try {
            Thread.sleep(SLEEP);

            if (this.circuitBreaker.isOpen()) {
                System.out.println("Do attempt execution #" + count.get());
            }

            if (count.getAndIncrement() <= THRESHOLD) {
                hasAddCount = true;
                System.out.println("Cause a failure #" + count.get());
                throw new Exception("Cause a failure");
            }

            System.out.println("success #" + count.get());
            return null;
        } finally {
            countDownLatch.countDown();
        }
    }

    @Override
    protected Void getFallback() {
        countDownLatch.countDown();
        if (!hasAddCount && count.getAndIncrement() % 50000 == 0) {
            System.out.println("Do fall back #" + count.get());
        }
        return null;
    }
}

At the beginning, the execution will be failed, but when count larger than THRESHOLD, the execution will be successful. And the attempt execution and fall back will also be successful.

In the concurrent scenario, with Hystrix 1.5.12, the final state of the execution will keep successful all the time. But with Hystrix 1.5.13, the state will be fail since the state of circuit breaker is open.

The reason should be that the attempt execution will markNonSuccess() in Hystrix 1.5.13 and make the attempt execution cannot set the state of circuit breaker from HALF_OPEN into CLOSE.