ci: add cron stress harness for serve with fault injection + metrics#427
Merged
tinder-maxwellelliott merged 3 commits intoJul 13, 2026
Merged
Conversation
tools/serve_stress.py complements the functional serve_harness.py by validating operational behavior under sustained load and injected faults, capturing throughput and response-time metrics for every request: - load phases: concurrent cache-hit bursts (hot), concurrent identical cold requests (checkout + query serialization), and a bad-request mix interleaved with good ones - forced git checkout errors: an orphaned index.lock planted before cold requests must self-heal to 200 (Tinder#425), and an index.lock storm during checkouts must only ever yield clean 200/400s with full recovery after - forced fetch errors: a git-daemon outage must fail fast with 400 and never lame-duck a ready instance, then recover once the remote returns; a fresh instance started during the outage must lame-duck (503 health, 503 queries, /metrics still served) - a background /health prober asserts readiness never regresses across every phase Per-phase throughput (req/s) and latency percentiles (p50/p90/p95/p99/max), time-to-ready, and /metrics snapshots are emitted as JSON (--metrics-out) and a markdown summary (--summary-out). The new serve-stress.yml workflow runs it on a daily cron (offset from serve-harness), publishes the summary to the step summary, and uploads the metrics JSON as a 90-day artifact for trend comparison across runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qd8xWWWRaXBULA192Ux9Dg
lock_heal asserted the "cleared stale git index.lock" self-heal line appears in serve's stderr, but that log goes through logger.w, which StderrLogger drops unless serve runs with -v. base.serve launched serve without -v, so the check failed every run (0 heals logged) and the job went red. Add an opt-in verbose_server flag to base.serve (default off, so the functional harness is unchanged) and enable it for the stress main instance; this also makes phase_final's git-checkout count accurate. Add real-repo mode (--real-repo-url/--real-from/--real-to/--real-clone-args): clone an actual repo, serve it with --no-initial-fetch, and drive real commit SHAs through concurrent cold/hot load plus the index.lock self-heal and storm phases, with invariant checks (200s, cross-request consistency, self-heal) rather than a hardcoded impacted set. Cold checkouts are forced with the base revision's real ancestry commits. Verified against bazelbuild/bazel (full bzlmod query, 5.2 MB/rev hash cache, bazel 9.1.1) and the bazel-diff repo. Add .github/workflows/serve-stress-real.yml: a 3-leg matrix (small=bazel-skylib, medium=abseil-cpp, bazel=bazelbuild/bazel) on a weekly cron + dispatch, each uploading per-size metrics for cross-size and over-time performance comparison. Kept off the daily hermetic cron since a full bazel query is minutes-scale. build_metrics_json now derives the phase list dynamically so real phase names surface. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Put all three legs (small/medium/bazel) on a daily cron (08:30 UTC, offset from serve-stress 30 6 and serve-harness 0 */12) alongside the other serve jobs, instead of the prior weekly schedule. Each leg keeps its 90-min timeout and fail-fast:false; the comment documents dialing back to weekly (30 8 * * 1) if a daily full bazel query is more runner cost than the trend data warrants. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
tinder-maxwellelliott
approved these changes
Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
tools/serve_stress.py complements the functional serve_harness.py by validating operational behavior under sustained load and injected faults, capturing throughput and response-time metrics for every request:
Per-phase throughput (req/s) and latency percentiles (p50/p90/p95/p99/max), time-to-ready, and /metrics snapshots are emitted as JSON (--metrics-out) and a markdown summary (--summary-out). The new serve-stress.yml workflow runs it on a daily cron (offset from serve-harness), publishes the summary to the step summary, and uploads the metrics JSON as a 90-day artifact for trend comparison across runs.
Claude-Session: https://claude.ai/code/session_01Qd8xWWWRaXBULA192Ux9Dg