Skip to content

test(benchmarks): align Polly comparisons - #138

Merged
thomhurst merged 1 commit into
mainfrom
fix/like-for-like-benchmarks
Aug 21, 2026
Merged

test(benchmarks): align Polly comparisons#138
thomhurst merged 1 commit into
mainfrom
fix/like-for-like-benchmarks

Conversation

@thomhurst

Copy link
Copy Markdown
Owner

Summary

  • align Kevlar and Polly circuit-breaker benchmarks on the same ratio, sampling, throughput, and break settings
  • compare manually isolated circuits on both sides and token-bucket rate limiters on both sides
  • make retry delays explicitly zero wherever only strategy machinery is intended to be measured
  • alternate and aggregate stress-test rounds so runner drift does not systematically penalize the second library
  • give corrected scenarios new history keys so previous non-equivalent measurements are not mixed into published medians

Motivation

The 15-minute stress run measured Kevlar about 5.6% below Polly despite the published microbenchmarks favoring Kevlar. The stress workload itself used matching ratio breakers, while the published composed benchmarks compared Kevlar's consecutive-failure breaker with Polly's ratio breaker. The rate-limit comparison also used Kevlar's token-bucket/GCRA limiter against Polly's fixed-window limiter, and the stress harness always measured Polly first and Kevlar second.

This change makes the compared semantics explicit and symmetric and balances early/late runner conditions.

Local benchmark check

BenchmarkDotNet short job, .NET 10.0.11, Windows, Intel Core Ultra 9 185H:

Scenario Kevlar Polly Allocation
Ratio breaker, closed 270 ns 315 ns 0 B / 24 B
Timeout → retry → ratio breaker 454 ns 1,236 ns 0 B / 48 B
Five-strategy token-bucket/ratio chain 572 ns 1,203 ns 0 B / 88 B
Token bucket, uncontended 152 ns 188 ns 0 B / 0 B

The short job is directional; the normal benchmark workflow remains the authoritative measurement.

Validation

  • dotnet build Kevlar.slnx -c Release
  • dotnet run --project tests/Kevlar.Tests -c Release --no-build -- --timeout 5m — 685 passed
  • dotnet run --project tests/Kevlar.AllocationTests -c Release --no-build -- --timeout 5m — 3 passed
  • BenchmarkDotNet dry validation — all 88 cases passed
  • BenchmarkDotNet corrected 13-case dry and short runs passed
  • 8-second alternating-round stress smoke test and both documentation generators passed

@thomhurst
thomhurst merged commit 7f9e060 into main Aug 21, 2026
5 checks passed
@thomhurst
thomhurst deleted the fix/like-for-like-benchmarks branch August 21, 2026 22:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant