Skip to content

v0.5.0

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 27 Jul 03:07
Immutable release. Only release title and notes can be modified.
v0.5.0
404c248
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Review precision: confidence now decides where a finding lands, newly-added parsers and regexes are stress-tested for their own failure modes, and several classes of false positive are guarded at both the generator and the verifier. Operators gain configurable CI-gating and blocking strictness, structured skip reasons, and per-model generation parameters.

Added

  • Mock-fidelity review check (#111): when a PR changes tests, the review prompt compares stubs/mocks (when/doThrow/doReturn) against the real collaborator's contract when that definition is already in the provided material, and flags contradictions (e.g. a mock that throws an exception the real method swallows) at low/medium confidence instead of treating the green test as proof. Softens the related-tests framing that previously reinforced unfaithful stubs; the skeptical verifier gains a matching rule. Broader cross-file collaborator retrieval remains with #55
  • Structured skip transparency for automatic reviews (#341): every path that skips an automatic review (draft, base-branch and label gates, /pause, rate window, duplicate webhook delivery, rejected dispatch) now emits a structured reason code — a Automatic review skipped [reason=...] log line, a thrillhouse.review.skips OTLP counter tagged with reason and repository, and per-reason counts on the dashboard summary endpoint (skippedReviewsByReason). README gains a "PR opened but no review posted" troubleshooting checklist
  • Bug-fix efficacy check (#110): when a PR declares itself a bug fix (the PR-template "Bug fix" checkbox or a Fixes/Closes/Resolves #N reference), the review prompt now extracts the concrete failure trigger from the PR description and the linked issues' text (fetched best-effort, up to 3 issues) and verifies the change actually alters behavior on that trigger's path — a locally-correct fix that the stated trigger never reaches is reported as a finding instead of passing silently, and when the deciding code is outside the diff the verdict is held at low confidence as a verification request rather than approved
  • Finding feedback capture for the learnings pipeline: records maintainer 👍/👎 reactions (and conservative "not useful" reply heuristics) on bot finding comments into a finding_feedback table, with per-repo aggregates on GET /api/dashboard/feedback and a documented data model/retention policy (docs/FEEDBACK.md). GitHub Apps have no reaction webhook, so reactions are polled via the Reactions API on review-thread replies and follow-up reviews. Feedback is permission-gated in both directions: only verified write-capable collaborators reacting on bot-authored finding threads are recorded, and GET /api/dashboard/feedback returns only repositories the authenticated user can access — so the endpoint cannot be used to enumerate activity on repos you are not a member of. Capture runs on a bounded executor that prioritizes newer findings and drops work rather than queueing without limit, and review logs no longer carry source-line contents at INFO. Precursor to cross-review learnings (#38); does not yet feed prompts (#324)
  • Adversarial failure-mode characterization for new heuristics (#123): when a diff introduces parsing, regex, validation, or heuristic code — a Pattern.compile, a tokenizer, a normalize step, a scope window or threshold constant — the reviewer switches those hunks from happy-path reading into decision-boundary characterization: it synthesizes the inputs that probe the rule's edges (a lambda against a paren-counting regex, an NBSP against an ASCII whitespace class, a token inside a fenced code block against a "starts with" check, a present-but-wrong-shape value against a presence check), names them literally, and reports false negatives as well as false positives. The triggering input is absent from the diff by definition, so the verifier gains a matching exemption from the verbatim-quote rule — without it the quote requirement rejects this entire class. Findings land at low/medium confidence as verification requests, and a synthesized failure is only confirmed when the expected domain or contract is visible in the material — a contract-free probe stays a low-confidence verification request. Executing the synthesized inputs is #96. The trigger spans languages: explicit regex construction (Java, JS/TS, Python, Go, C#, Kotlin), JS/TS regex literals and function/arrow validators, declared parse/validate/tokenize members including package-private Java methods, and construction split across lines via a bounded, file-reset-aware window. It stays deliberately narrow so ordinary substring/indexOf/trim/parseInt code does not trigger it (dogfood: three review rounds on PR #114 returned "no issues" while human review found ~10 precision/recall defects in exactly this kind of code)
  • Configurable CI gating strictness for APPROVE (#322): REVIEW_CI_GATING chooses how CI status factors into approval — strict (default) holds APPROVE while required CI is pending, failing, or unreadable; warn allows APPROVE but records the CI uncertainty in the summary and check run; off decides on findings alone
  • Configurable blocking strictness (#323): REVIEW_BLOCKING_STRICTNESS sets when findings escalate to REQUEST_CHANGES — balanced (default: CRITICAL/HIGH at high confidence), strict (any CRITICAL/HIGH), or lenient (CRITICAL at high confidence only), so a repo can tune how loudly the bot blocks without editing prompts
  • Model generation parameters (#64): frequency-penalty, presence-penalty and seed join the existing per-model settings under thrillhousebot.ai.models.<model>.*, sent on every chat call when set. seed makes a provider's sampling reproducible where supported, which is what lets the #113 eval corpus compare runs meaningfully

Changed

  • Passing in-diff tests must exercise the claimed path before suppressing a finding (#116): the review and verifier prompts no longer treat a green test in the same diff as automatic disproof. A test may invalidate (generator) or reject (verifier) a finding only when it demonstrably exercises the claimed path — asserting on the path's output and stubbing collaborators into the relevant state, not leaving them unmocked so a default bypasses it (and not with stubs that contradict the real collaborator — see #111). When that exercise cannot be shown from the provided material, confidence is lowered with an explanation instead of dropping the finding (dogfood: PR #99 approve-path test green because getPullRequest was unmocked)
  • Low-confidence findings post in the summary instead of inline (#105): inline review comments are now gated on confidence — a low-confidence finding is listed in the summary body rather than anchored to a line. Uncertain claims still surface, but they stop occupying the position that reads as "this line is wrong", and they can no longer request changes on their own
  • Exact-arithmetic and "this test fails" claims are capped and hedged (#97): a finding that rests on line-count, array-length, or index arithmetic the model performed by counting, or that asserts a specific test fails, is now capped at low confidence when nothing but counting backs it, and must be phrased as a verification request ("CI will confirm") rather than settled fact. The verifier gains the symmetric rule and rejects such a claim stated as fact with no execution or CI signal behind it — it cannot settle the question by recounting, being the same modality reading the same diff. The cap is evidence-dependent, not absolute: when an execution or CI signal in the provided material actually shows the failure, the finding keeps the confidence that evidence justifies instead of being forced low. Mirrors the existing "why would this test still pass" guard for the failure direction (dogfood: PR #84 produced a definitively-worded off-by-one — "the section is 7 lines, so 7 − 3 = 4 omitted" — against a test that passes as written, and the verifier re-did the same arithmetic and agreed)
  • Dual-gate merge policy for contributors (#342): merging now requires both the static analysis gate and a ThrillhouseBot review, so neither signal alone can carry a change in. Affects anyone opening a PR against this repo rather than anyone running the bot
  • Wider default ignored-files patterns (#52): thrillhousebot.review.ignored-files now skips far more generated and vendored output out of the box — pnpm-lock.yaml, go.sum, protobuf output (*.pb.go, *_pb2.py), minified bundles and sourcemaps (*.min.js, *.min.css, *.map), and the directories node_modules/, dist/, build/, out/, .next/, vendor/, __pycache__/, .venv/, bin/, obj/, alongside the previous lockfile/pom.xml/target/ set. Less budget burned on artifacts nobody reviews. Worth checking on upgrade: the directory globs match by name, so a repository that keeps handwritten source under build/, bin/, out/ or vendor/ will stop having it reviewed — narrow the list per deployment if that applies to you

Fixed

  • False parameter-nullability / precondition findings when the caller is outside the diff: a finding could assert a method parameter may be null (or violates a precondition) and post at MEDIUM even when the only callers that supply the argument were unchanged and absent from the reviewed material — so nullability was assumed rather than demonstrated (a MEDIUM false positive on DashboardAccessChecker.installedRepos in PR #101 claimed accountOwner could be null while the unshown checkAccess/evaluateAccess callers already guarantee non-null). The reviewer now caps such claims to low confidence unless the calling code is present and shown to pass a violating value (or the changed signature itself declares a nullable contract such as @Nullable / Optional), and the verifier rejects them when neither the caller nor such a contract is in the material — with the PR #101 case embedded as an inline regression example (#107)
  • Findings survived a force-push that removed the lines they targeted (#336): when a rebase or amend made a targeted diff hunk disappear, prior findings anchored there were still carried forward and re-reported against code that no longer existed. Presence is now re-checked against the current diff and findings whose anchors are gone are invalidated instead of re-raised
  • False injection and constructor findings on framework-idiomatic code (#337): the reviewer flagged idiomatic framework usage as broken — most visibly a "missing no-arg/default constructor" on a CDI/Quarkus bean whose constructor is annotated @Inject (and the Spring @Autowired equivalent), where constructor injection is the documented idiom and needs no such constructor. The generator and verifier now recognise these shapes and refuse the claim rather than posting it
  • Duplicate CI-pending comment when the summary already listed the pending checks (#334): a follow-up COMMENT review announcing pending CI could post alongside a summary that already enumerated the same checks, telling maintainers the same thing twice. The extra comment is now suppressed unless the summary did not actually post

Performance

  • Pure renames no longer consume AI review budget (#386): a rename with no content change contributes an empty hunk, so it was spending token budget while carrying no reviewable text. Renamed files are excluded from the reviewable diff (still passed to the related-tests and mock-fidelity context so test moves stay visible), leaving more of the budget for files that actually changed
  • Deduplicated work on the follow-up backstop hot path (#135): the #118 backstop rebuilt the DiffLineResolver, re-parsed prior AI responses, and copied the status list more than once per review. Each is now built once and shared across the pass

Dependencies

  • Bumped the Quarkus platform group, the frontend npm-minor-patch group, @testing-library/jest-dom to 7.0.0, the website docs-minor-patch group, dompurify to 3.4.12, svgo to 4.0.2, and the GitHub Actions minor/patch group
  • The docs site now requires Node 22 and pins PostCSS 8.5.23 and Sharp 0.35.3, so a clean npm ci on npm 11 resolves without known high-severity advisories

What's Changed

Features

Fixes

Other changes

New Contributors

Full Changelog: v0.4.0...v0.5.0

Contributors

devops-thiago and matheusandre1
Assets 10
Loading