Drive the circuit breaker's clock from tests, not sleep() - #619
Merged
Conversation
The circuit breaker already had the seam — every time read goes through CircuitBreaker.now() — it just hardcoded time.Now(). Give it an unexported nowFn defaulting to time.Now, so in-package tests can substitute a clock they advance by hand. TestCircuitBreakerResetsStaleHalfOpenAttempts is the test that flaked: it slept against a 100ms StaleAttemptTimeout, and under -race on a loaded runner the stale reset could fire early. Sleeping to cross a timeout makes the assertion a bet on scheduler latency. Convert every sleeping circuit-breaker test, not just that one, and use realistic production timeouts now that crossing them is free: - ClosesAfterSuccesses, StateTransitionsCorrectly, ResetsStaleHalfOpenAttempts - SetsHalfOpenLastAttemptAt, whose real-time before/after window assertion is meaningless once the clock is fake — it becomes an exact equality against the Allow() that reserved the slot - The hooks integration test, which needs no new seam: NewGatingHooks already takes the breaker, so it builds its own and passes it in `go test -race -count=20 ./internal/resilience/` is stable. The package is not sleep-free — rate_limiter_test.go has three sleeps of its own against a primitive with no now() seam, and those are untouched here. Closes #586
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The circuit breaker already had the seam — every time read goes through
CircuitBreaker.now()— it just hardcodedtime.Now(). This gives it anunexported
nowFndefaulting totime.Now, so in-package tests can substitutea clock they advance by hand. The constructor signature is unchanged; the only
non-test caller (
hooks.go:36) is untouched.TestCircuitBreakerResetsStaleHalfOpenAttemptsis the reported flake: it sleptagainst a 100ms
StaleAttemptTimeout, so under-raceon a loaded runner thestale reset could fire early. Sleeping to cross a timeout makes the assertion a
bet on scheduler latency.
Every sleeping circuit-breaker test is converted, not just that one — a fake
clock next to four other tests still racing real time would just relocate the
flake. With crossing a timeout now free, they also use realistic production
timeouts (30s open, 2min stale) instead of the millisecond values that only
existed to keep the sleeps short:
ClosesAfterSuccesses,StateTransitionsCorrectly,ResetsStaleHalfOpenAttemptsSetsHalfOpenLastAttemptAt— its real-time.Now()before/after windowassertion becomes meaningless (and separately flaky) once the clock is fake,
so it becomes an exact equality against the
Allow()that reserved the slothooks_test.go's integration test needs no new seam:NewGatingHooksalreadytakes the breaker, so the test builds its own, sets
nowFn, and passes it inVerified:
go test -race -count=20 ./internal/resilience/is stable, andbin/ciis green.Not claimed: the package is not sleep-free.
rate_limiter_test.gohasthree sleeps of its own against a separate primitive with no
now()seam. Theyare untouched, so package runtime does not drop to zero — the claim here is
about the converted tests only.
Closes #586
Summary by cubic
Inject a test-driven clock into the circuit breaker via an unexported
nowFn, removing sleeps from tests and fixing the half-open stale attempt flake. No public API changes.nowFntoCircuitBreaker(defaults totime.Now) and routed all time reads through it.SetsHalfOpenLastAttemptAtto assert the exact timestamp set byAllow().go test -race -count=20 ./internal/resilience/;rate_limiter_test.gosleeps remain. Closes Flaky under -race: TestCircuitBreakerResetsStaleHalfOpenAttempts #586.Written for commit 069a5d6. Summary will update on new commits.