Add #206 benchmark methodology and #207 cache-key derivation to the roadmap - #208
Merged
Conversation
#88 merged, so #171 becomes the next slice. #206 comes from validating #88: four smoke suites ran against a gate that never benchmarks list_projects, host variance ran 3-4x uncorrelated with load, and the pass/fail flipped twice on noise while the question was finally answered by a ten-minute interleaved A/B. It covers coverage checks, baseline comparison, explicit better/worse/inconclusive criteria, and a merge recommendation. Sequenced right after #171 so its approach can be exercised while benchmarking #171 and refined from that experience. #207 records a near miss rather than a live bug. CacheKey and the shouldBypassCache list agree by inspection, not construction; #88 nearly shipped a rootOnly key omission that would have served filed projects to an unfiled-projects query with no error and no warning. Adds a standing decision that a change is benchmarked on the path it changed, and that a single stalled call is an observation rather than a verdict. Validation impact: docs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Docs-only. Impact:
docs(validation passes).#88 merged, so #171 becomes the next slice.
#206 comes directly from validating #88: four 10-minute smoke suites ran against a gate that never benchmarks
list_projects, host variance ran 3-4x uncorrelated with load, and the pass/fail flipped twice on noise — while the actual question was answered by a ten-minute interleaved A/B. It covers benchmark coverage checks, interleaved baseline comparison, explicit better/worse/inconclusive criteria, and an emitted merge recommendation. Sequenced immediately after #171 so its approach gets exercised while benchmarking #171, then refined from that experience.#207 records a near miss, not a live bug:
CacheKeyand theshouldBypassCachelist agree by inspection rather than construction, and #88 nearly shipped arootOnlyomission that would have served filed projects to an unfiled-projects query with no error.Also adds a standing decision: benchmark a change on the path it changed, and treat a single stalled call as one observation rather than a verdict.
🤖 Generated with Claude Code