chore: add quality gates for agent-authored changes - #19
Merged
Conversation
Adds the guardrails that keep this repo clean as more of its changes are written by coding agents. Targets what the 2026 research on AI-authored code actually measures degrading — duplication (+81% since 2023), collapsed refactoring (-70%), and error-masking (+47%) — rather than dead code alone, which is mostly the downstream symptom. Blocking gates: - CLAUDE.md: the conventions and invariants an agent cannot infer from the source. This is the only guardrail that acts before code is written rather than rejecting it after. - tests/test_invariants.py: five architectural rules enforced as tests — credentials resolved only in shared/auth.py, subprocess only via run_cli, no shell=True, config path not rebuilt, MCP route maps stay read-only. Each was verified to fail when violated. These encode this package's design, which no off-the-shelf linter can check. - Diff coverage at 80% via pytest-cov + diff-cover, gating lines the PR changed rather than the repo total. A total floor would punish unrelated work and invite gaming; total coverage is 74%, concentrated in the two surfaces that are thinnest by history, not by this PR's doing. - Ruff: +ARG (unused arguments), +BLE (blind except), +TRY, +C90 (complexity <=10), +ERA (commented-out code). TRY003/TRY004 are ignored with reasons — both fight deliberate conventions in this package. - Ruff pinned exactly (==0.15.13) and .pre-commit-config.yaml's rev bumped to match. They had drifted five minor versions apart, so pre-commit could format code that CI would then reject. Advisory (reports to the job summary, does not block): - vulture for dead public symbols, jscpd for duplicated blocks. Duplication baseline is 0 clones. Per the phased approach, these stay non-blocking until we have a few weeks of real trend data. The one existing blind `except` is now annotated with its rationale and covered by a test, rather than silently suppressed. vulture already found real drift: Settings.tool_timeout and Settings.max_tool_output are documented env-var knobs that nothing reads — every tool calls run_cli() without them, so _shell.py's module defaults win. Left unwhitelisted so the report keeps surfacing it; wiring them through is a separate change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FLcFuAB6HLYNEMVoEYaZgw
The vulture and jscpd steps wrote solely to $GITHUB_STEP_SUMMARY, so their output never appeared in the job log. Combined with the `|| true` that keeps the advisory job green, a tool that failed to run at all looked exactly like a tool reporting a clean result — the first run could not be verified as having actually executed jscpd. Each report now goes to a file that is both catted to the log and appended to the summary. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FLcFuAB6HLYNEMVoEYaZgw
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the guardrails that keep this repo clean as more of its changes are written by coding agents.
Aimed at what the 2026 research on AI-authored code actually measures degrading — duplication (+81% since 2023), collapsed refactoring (−70%), error-masking constructs (+47%) — rather than dead code alone, which is mostly the downstream symptom of the first two.
Blocking gates
CLAUDE.mdtests/test_invariants.pyshared/auth.py,subprocessbypassingrun_cli,shell=True, rebuilt config paths, MCP route maps losing read-only-ness.pytest-cov+diff-cover.+ARG +BLE +TRY +C90 +ERA==0.15.13On the invariant tests
Each of the five was verified by introducing the violation and confirming it fails — and that it fails only its own test. These encode design rules specific to this package that no off-the-shelf linter can express, which is the class of rule that otherwise erodes one reasonable-looking PR at a time.
On diff coverage rather than total
Total coverage is 74%, concentrated in
mcp/server.py(0%) andsre/cli/app.py(28%) — thin by history, not by anything in this PR. A repo-wide floor would either block unrelated work or have to be set so low it gates nothing. A diff floor asks exactly one thing: test what you just wrote.On the ignored rules
TRY003andTRY004are ignored with written reasons. Both fight deliberate conventions here: an exception's message is the user-facing text (AuthError), and the isinstance guards in the SDK validate an unexpected wire-response shape rather than a caller's argument type.The one pre-existing blind
except(_load_settings) is now annotated with its rationale and covered by a test, rather than silently suppressed.Advisory (reports to the job summary, does not block)
vulturefor unused public symbols andjscpdfor duplicated blocks — the two signals ruff and pyright structurally cannot see. Duplication baseline is 0 clones. These stay non-blocking until there are a few weeks of trend data.Real drift this already found
Settings.tool_timeoutandSettings.max_tool_outputare documented env-var knobs (YERTLE_SRE_TOOL_TIMEOUT,YERTLE_SRE_MAX_TOOL_OUTPUT) that nothing reads — every tool callsrun_cli(argv)bare, so_shell.py's module defaults win. The values coincide (30 / 10_000), so behavior is correct by accident; only the overrides are inert. Left unwhitelisted so the report keeps surfacing it. Wiring them through has no natural injection point in the module-level@toolfunctions, so it's a design change and deliberately out of scope here.ShellResult.truncatedis the same shape: set, never read by any caller.Verification
make checkgreen — 99 tests, diff coverage 100%.🤖 Generated with Claude Code
https://claude.ai/code/session_01FLcFuAB6HLYNEMVoEYaZgw