feat(selfhost): PostHog AI observability across ORB and AMS#8623
Conversation
Captures $ai_generation/$ai_embedding events for every self-host AI provider attempt (the AI_PROVIDER chain plus the AI_EMBED/AI_VISION/ AI_ADVISORY bindings) and for every AMS coding-agent driver attempt (claude-cli/codex-cli/agent-sdk), metadata only -- model, provider, latency, tokens, cost, never prompt/response content. Adds the miner CLI's own opt-in PostHog client (packages/loopover-miner/ lib/posthog.ts) as the foundation the AMS side needs, running alongside the existing Sentry integration during the parallel-run window. Includes egress-allowlist entries, env-reference regeneration, and DEPLOYMENT.md updates for the two new miner env vars.
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8623 +/- ##
==========================================
- Coverage 92.76% 90.41% -2.36%
==========================================
Files 797 103 -694
Lines 79355 23434 -55921
Branches 24034 4221 -19813
==========================================
- Hits 73616 21187 -52429
+ Misses 4592 2031 -2561
+ Partials 1147 216 -931
Flags with carried forward coverage won't be shown. Click here to find out more.
|
loopover-orb
Bot
added
gittensor:bug
|
Tip ✅ LoopOver review result - approve/merge recommendedReview updated: 2026-07-25 15:23:45 UTC
Review summary Nits — 6 non-blocking
Decision drivers
Context & advisory signals — never blocks the verdict
Linked issue satisfactionAddressed Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. Visual preview
Click any thumbnail to open the full-size screenshot. Before = production · After = this PR's preview deploy. Scroll preview
A short scroll-through clip (desktop) — click either thumbnail to open the full animation. Evidence for scroll-linked behavior a single screenshot can't show. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
![loopover-orb[bot]](https://avatars.githubusercontent.com/in/4139483?s=60&v=4){kind=small}
1 participant
Summary
AI_PROVIDERchain — ollama/openai/openai-compatible/anthropic/claude-code/codex — plus the three standaloneAI_EMBED/AI_VISION/AI_ADVISORYbindings) now emits a PostHog$ai_generation($ai_embeddingfor embed requests) event: model, provider, latency, tokens, cost (when derivable), effort, and repo/PR correlation — metadata only, never prompt/response content. Wired at the single real chokepoint (runProviderWithOtel,src/selfhost/ai.ts) plus a newwithAiGenerationCapturewrapper for the three ungoverned bindings.packages/loopover-miner/lib/posthog.ts(mirrorssentry.ts's contract, runs alongside it per the epic's parallel-run posture) plus$ai_generationcapture for all three coding-agent driver types (claude-cli/codex-cli/agent-sdk), wired atconstructProductionCodingAgentDriver. Honestly represents the data it actually has:CodingAgentDriverResultonly reports a single blendedcostUsd/tokensUsed(no input/output split like ORB'sAiUsage), so the new capture never fabricates a split — token count rides in a plaintokens_usedproperty,$ai_total_cost_usd(one of PostHog's own recognized fields, needs no split) is populated directly._FILEsecret-mount support (free via the existing generic<NAME>_FILEscanner), env-reference regeneration, andDEPLOYMENT.mdupdated for the two new miner env vars.Scope
type(scope): short summaryConventional Commit format.CONTRIBUTING.mdand does not reintroduce GitHub Pages, VitePress,site/, orCNAME.Closes #8296).Validation
git diff --checknpm run actionlintnpm run typechecknpm run test:coveragelocally — full unsharded run: 21,576 tests passed, 0 failed. Every changed file at 100% line+branch coverage on the diff (verified via scoped v8 runs); the two pre-existing sub-100% lines inai.ts/posthog.ts(ORB) are outside this diff's hunks, confirmed viagit diffline-range comparison.npm run test:workersnpm run build:mcpnpm run test:mcp-packnpm run build:minernpm run test:miner-packnpm run test:miner-deployment-docs-auditnpm run miner:env-reference:checknpm run ui:openapi:checknpm run ui:lintnpm run ui:typechecknpm run ui:buildnpm run docs:drift-checknpm run engine-parity:drift-checknpm audit --audit-level=moderate— 5 pre-existing high-severity findings, all transitive dev-tooling (eslint/minimatch/brace-expansion); zero new dependencies added by this PR (package.json/package-lock.jsonuntouched).If any required check was skipped, explain why:
npm run test:cichain was run end-to-end on the final, rebased tree.Safety
self-hosting-operations.mdx) is prose only, no new UI component.UI Evidencesection — not applicable, no visible UI/frontend change.UI Evidence
Not applicable — backend + miner-CLI observability instrumentation and doc prose only, no visible UI/frontend/extension change.
Notes
selfhost-posthog.test.ts) and AMS (miner-posthog.test.ts) capture functions asserting$ai_input/$ai_output_choicesnever appear.vectorize.ts/pg-vectorize.ts/qdrant-vectorize.ts) are deliberately out of scope — they store/query already-computed embedding vectors and never call an AI model themselves; the actual embedding generation call (which does call a model) already flows through the same instrumented chokepoint as chat/review calls._FILE-secret/DEPLOYMENT.mddeliverables are included in full as the necessary foundation for the AMS half; Sentry stays fully functional in parallel per the epic's own parallel-run posture (both capture when both configured).