Skip to content

fix(observability): capture AI-CLI stderr + surface exhausted provider failures to Sentry#1605

Merged
JSONbored merged 2 commits into
mainfrom
obs/ai-stderr
Jun 27, 2026
Merged

fix(observability): capture AI-CLI stderr + surface exhausted provider failures to Sentry#1605
JSONbored merged 2 commits into
mainfrom
obs/ai-stderr

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

The self-host claude/codex review CLIs intermittently fail with an opaque claude_code_exit_1 / codex_exit_1, and provider outages never surface in Sentry. Two gaps:

  1. defaultSpawn captured stdout but discarded stderr — where the CLI's real error (auth failure, rate limit, model-not-supported, OOM) actually lands. The exit code alone is undiagnosable.
  2. Per-attempt provider failures log at warn; the central Sentry forwarder only ships error/fatal, so a genuine reviewer outage is invisible.

Changes

  • src/selfhost/ai.tsdefaultSpawn now captures the child's stderr and threads it into the claude_code_exit_* / codex_exit_* error message (truncated to 500 chars). The stderr field is optional on SpawnFn so existing spawn stubs are unaffected.
  • src/services/ai-review.tsrunWorkersOpinion emits one ai_review_provider_exhausted log at error level once all models × attempts have thrown (the reviewer is genuinely down). The noisy per-attempt retries stay at warn. A model that runs but returns unparseable output is not an exhaustion (no provider error) and stays silent.

Testing

  • defaultSpawn captures a failing CLI's stderr (real subprocess) and surfaces it on the exit error.
  • stderr appears in both the claude and codex exit errors (mocked).
  • ai_review_provider_exhausted fires at error level when every attempt throws; does not fire on ran-but-unparseable output.
  • Affected suites green (152 tests); patch coverage 100% on changed lines.

Advances #26 (comprehensive Sentry instrumentation — surface all failures).

…r failures to Sentry

The claude/codex review CLIs intermittently failed with an opaque claude_code_exit_1 that was undiagnosable: defaultSpawn discarded stderr, and per-attempt provider failures logged at warn level, which the central Sentry forwarder (error/fatal only) skips.

- defaultSpawn now captures the child stderr and threads it into the exit error, so the real cause (auth, rate limit, model-not-supported) reaches the logs.
- runWorkersOpinion emits one ai_review_provider_exhausted log at error level once all models and attempts have thrown, so a genuine reviewer outage surfaces in Sentry while the noisy per-attempt retries stay at warn.
@dosubot dosubot Bot added the size:M label Jun 27, 2026

@superagent-security superagent-security Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superagent found 1 security concern(s).

Comment thread src/selfhost/ai.ts Outdated
@superagent-security superagent-security Bot added the pr:flagged PR flagged for review by security analysis. label Jun 27, 2026
@codecov

codecov Bot commented Jun 27, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.51%. Comparing base (bd59608) to head (20f47ac).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1605   +/-   ##
=======================================
  Coverage   95.50%   95.51%           
=======================================
  Files         204      204           
  Lines       22018    22030   +12     
  Branches     7957     7959    +2     
=======================================
+ Hits        21029    21041   +12     
  Misses        413      413           
  Partials      576      576           
Files with missing lines Coverage Δ
src/selfhost/ai.ts 99.44% <100.00%> (+0.02%) ⬆️
src/services/ai-review.ts 98.64% <100.00%> (+0.01%) ⬆️
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JSONbored JSONbored self-assigned this Jun 27, 2026
@loopover-orb

loopover-orb Bot commented Jun 27, 2026

Copy link
Copy Markdown

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review — held for maintainer review

4 files · 1 AI reviewers · no blockers · readiness 66/100 · CI pending · blocked

⏸️ Held for maintainer review — Touches a guarded path — held for manual review

Signal Result Evidence
Code review ✅ No blockers 1 reviewers, synthesized
Linked issue ⚠️ Missing No linked issue or no-issue rationale found.
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Review load ✅ 20/20 Readiness component derived from cached public PR metadata and labels; size label size:M.
Validation evidence ❌ 5/25 Cached preflight status is hold.
Open PR queue ❌ 3/10 24 open PR(s), 10 likely reviewable, 14 unlinked.
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 80 PR(s), 301 issue(s).
Gate result ⚠️ Not blocking Advisory; not blocking this PR.
Nits — 1 non-blocking
  • Touches a guarded path — held for manual review — A maintainer must review and merge this change.
Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository registration is not available in the local Gittensory cache.
  • Public profile languages: not available
  • Official Gittensor activity: 80 PR(s), 301 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Treat this as maintainer-lane context rather than normal contributor-lane activity.
  • Explain no-issue PR.
  • Fix blocker.
  • Expect slower review.
  • Refresh registry data or choose a registered active repo.
  • Link the issue being solved, or explicitly explain why this is a no-issue PR.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Review load = cached public PR metadata such as size labels, changed paths, and preflight status.
  • Open PR queue = repo-wide review pressure; it is not a PR quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@loopover-orb loopover-orb Bot added gittensor gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. labels Jun 27, 2026
…errors

The prior commit embedded raw claude/codex subprocess stderr into the thrown exit error. That stderr can echo the OAuth token we pass via env (or a key from a config the CLI reads), and the error flows to logs and the Sentry forwarder, whose scrubber only redacts secret-keyed fields, never free-text -- so a token inside the error string would leak.

Add redactSecrets(): strip the caller's known secret values exactly, then well-known credential shapes (sk-/sk-ant-/sk-proj-, GitHub PAT/OAuth/server tokens, JWT, AWS key id), each word-boundary anchored so genuine diagnostics survive. Applied at both the claude (with the OAuth token) and codex throw sites before truncation.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@superagent-security superagent-security Bot removed the pr:flagged PR flagged for review by security analysis. label Jun 27, 2026
@JSONbored
JSONbored merged commit 468f00d into main Jun 27, 2026
19 checks passed
@JSONbored
JSONbored deleted the obs/ai-stderr branch June 27, 2026 16:36
JSONbored added a commit that referenced this pull request Jun 28, 2026
…the event slug (#1636)

forwardStructuredLogToSentry titled each issue with only obj.event and buried
the real failure in a 'log' context blob — so operators had to open each issue
to learn what broke. Now:
- title leads with the failure: "<event>: <message ?? error>" (e.g.
  orb_broker_unavailable: The operation was aborted due to timeout)
- index filterable tags for the dimensions operators search/group by (repo,
  installationId, pull, pr, project, kind, deliveryId), present values only
- fingerprint by event so recurrences collapse into one issue instead of
  fragmenting on the variable detail now in the title

Pairs with the source-side instrumentation (#1605/#1619/#1625/#1627/#1628):
those make failures reach Sentry; this makes each issue legible at a glance.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant