Skip to content

feat(review): expose real BYOK token/cost usage in ops stats#3992

Merged
JSONbored merged 1 commit into
mainfrom
feat/byok-usage-observability
Jul 7, 2026
Merged

feat(review): expose real BYOK token/cost usage in ops stats#3992
JSONbored merged 1 commit into
mainfrom
feat/byok-usage-observability

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • Closes feat(observability): expose real ai_usage_events token/cost data for the hosted Worker deployment #3925 (rescoped after clarifying with the user: the hosted Worker's legacy Workers-AI-binding path is fully retired -- env.AI is undefined, no binding in wrangler.jsonc -- so BYOK is the ONLY AI activity a hosted deployment can ever have; currently zero repos have a BYOK key configured, so this reports all zeros today).
  • Migration 0109 added real input_tokens/output_tokens/total_tokens/cost_usd columns to ai_usage_events, but nothing read them back for the hosted deployment -- the one dashboard built for this (orb-ai-usage.json) is wired exclusively to self-host's own local reporting-export SQLite mirror and cannot see the hosted D1's rows.
  • Adds sumByokAiUsageForRepoSince (real, not estimated tokens/cost, trailing 24h window) and wires it into the existing /v1/internal/ops/stats payload as a new byokUsage field per repo.

Scope

  • Conventional Commit title, focused single change.
  • No issue-link checkbox in the template sense -- Closes #3925 above.

Validation

  • npm run typecheck
  • npx vitest run test/unit/ops-wire.test.ts -- 28/28 pass, including a real-usage rollup test and a zero-usage negative test.
  • 100% branch coverage on the changed lines (verified via lcov).

Safety

  • No secrets/private data (aggregate counts/tokens/cost only, no PR content).
  • No auth/API-shape changes to the existing bearer-gated endpoint, additive field only.

Closes #3925. The hosted gittensory-api Worker's ONLY possible AI
activity is a maintainer's own BYOK call (the legacy Workers-AI
binding path is retired; env.AI is undefined there) -- but nothing
previously read back the real token/cost columns migration 0109 added
to ai_usage_events for the hosted deployment. The one dashboard built
for this (orb-ai-usage.json) is wired exclusively to self-host's own
local reporting-export SQLite mirror, which cannot see the hosted D1.

Adds sumByokAiUsageForRepoSince (real, not estimated, tokens + cost
over a trailing 24h window) and wires it into the existing
/v1/internal/ops/stats payload as a new byokUsage field per repo.

Currently zero repos have a BYOK key configured, so this reports all
zeros today -- the point is that a future burn is now observable at
all, rather than requiring an ad hoc manual D1 query.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@JSONbored JSONbored self-assigned this Jul 7, 2026
@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 7, 2026
@JSONbored
JSONbored merged commit b414e5e into main Jul 7, 2026
6 of 8 checks passed
@JSONbored
JSONbored deleted the feat/byok-usage-observability branch July 7, 2026 09:31
@codecov

codecov Bot commented Jul 7, 2026

Copy link
Copy Markdown

❌ 4 Tests Failed:

Tests completed Failed Passed Skipped
11140 4 11136 7
View the full list of 4 ❄️ flaky test(s)
test/unit/moderation-config-db.test.ts > global moderation config DB round-trip (#selfhost-mod-engine) > REGRESSION (gate-flagged): violationDecayDays above MAX_MODERATION_VIOLATION_DECAY_DAYS is CLAMPED, not passed through raw -- an unbounded value overflows Date arithmetic on the live close path

Flake rate in main: 100.00% (Passed 0 times, Failed 1 times)

Stack Traces | 0.396s run time
AssertionError: expected 100 to be 3650 // Object.is equality

- Expected
+ Received

- 3650
+ 100

 ❯ test/unit/moderation-config-db.test.ts:91:41
test/unit/moderation-config-db.test.ts > global moderation config DB round-trip (#selfhost-mod-engine) > a raw DB row with an over-max violation_decay_days is also clamped on READ (not just on write)

Flake rate in main: 100.00% (Passed 0 times, Failed 1 times)

Stack Traces | 0.372s run time
AssertionError: expected 100 to be 3650 // Object.is equality

- Expected
+ Received

- 3650
+ 100

 ❯ test/unit/moderation-config-db.test.ts:108:41
test/unit/moderation-config-db.test.ts > global moderation config DB round-trip (#selfhost-mod-engine) > a violationDecayDays AT the max is preserved unclamped (boundary, not just strictly-under)

Flake rate in main: 100.00% (Passed 0 times, Failed 1 times)

Stack Traces | 0.378s run time
AssertionError: expected 100 to be 3650 // Object.is equality

- Expected
+ Received

- 3650
+ 100

 ❯ test/unit/moderation-config-db.test.ts:101:41
test/unit/queue.test.ts > queue processors > a #1960 action-command verb with no dispatch handler wired yet (e.g. pause) is bailed out of the Q&A answer-card path, not misrendered as help (#2160)

Flake rate in main: 100.00% (Passed 0 times, Failed 1 times)

Stack Traces | 0.881s run time
AssertionError: expected 1 to be +0 // Object.is equality

- Expected
+ Received

- 0
+ 1

 ❯ test/unit/queue.test.ts:22807:28

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(observability): expose real ai_usage_events token/cost data for the hosted Worker deployment

1 participant