selfhost(telemetry): Gittensory Orb — outcome signal sidecar + calibration loop

[JSONbored]

Part of #1029.

Context

Today the gate makes a decision (merge / close / block) and the signal dies. There is no way to know: did the merged PR introduce a bug that was reverted? Did the review comment actually get addressed? Did the closed PR's author come back and ship it correctly? Without outcome data, the gate's thresholds, the AI prompts, and the time-decay model are all calibrated on guesses. Gittensory Orb closes that loop.

Orb is an opt-in telemetry sidecar — a separate GitHub App (not the review App) that gets installed on repos by operators who want to participate in the calibration program. It collects outcome-only, metadata-only signals and feeds them back to a central calibration service (or a self-hosted collector for air-gap operators). This dataset is the long-term moat: every review decision becomes a labeled training sample.

Easy on/off

# Enable:
docker compose --profile orb up -d
# Add to .env:
# ORB_GITHUB_APP_ID=...
# ORB_GITHUB_PRIVATE_KEY=...
# ORB_COLLECTOR_URL=https://orb.gittensory.app   # or http://your-collector:9000 for air-gap

# Disable:
# Remove the ORB_* vars and stop the profile. Zero impact on the review engine.

The review engine has no code dependency on Orb — it runs in a parallel container, listens on its own webhook endpoint, and writes to its own table.

What Orb collects (metadata only — NO code content)

Signal	How collected	Privacy
PR merge / close / revert	GitHub pull_request events	No diff, no code
Time-to-merge distribution	Timestamps from merge events	Aggregate only
Review comment acceptance rate	pull_request_review + subsequent push events	Whether the flagged file was modified — no content
Revert detection	push events where commit message contains revert + matches a prior PR SHA	Commit SHA + flag only
Gate verdict at merge-time	Pulled from local audit_events ledger	Already in the DB

Nothing that would identify contributor identity, expose diffs, or leak business logic.

Architecture

Orb GitHub App

• A separate GitHub App with pull_request: read, checks: read permissions only
• Installed independently of the review App — operators install it on repos they want to contribute outcome data from
• Sends events to POST /orb/webhook on the self-host container

docker-compose.yml

orb:
  image: gittensory-selfhost-gittensory   # same image, different entry
  profiles: ["orb"]
  command: ["node", "dist/orb.mjs"]       # dedicated orb entry
  restart: unless-stopped
  environment:
    ORB_GITHUB_APP_ID: ${ORB_GITHUB_APP_ID}
    ORB_GITHUB_PRIVATE_KEY: ${ORB_GITHUB_PRIVATE_KEY}
    ORB_GITHUB_WEBHOOK_SECRET: ${ORB_GITHUB_WEBHOOK_SECRET}
    ORB_COLLECTOR_URL: ${ORB_COLLECTOR_URL:-https://orb.gittensory.app}
    ORB_INCLUDE_REVIEW_VERDICTS: ${ORB_INCLUDE_REVIEW_VERDICTS:-true}
    DATABASE_URL: ${DATABASE_URL:-}         # reads audit_events from the same DB
  depends_on:
    gittensory:
      condition: service_healthy
  ports:
    - "8788:8788"

src/selfhost/orb.ts (new entry)

• Validates the Orb App webhook signature
• On pull_request.closed (merged=true): emits {event: "orb_merge", repo, pr, sha, verdict: getVerdictFromAuditLog()}
• On push with revert keyword: emits {event: "orb_revert", repo, pr_sha}
• Batches signals, signs with installation public key, posts to ORB_COLLECTOR_URL
• Air-gap mode: writes to orb_events table if ORB_COLLECTOR_URL=local

Privacy controls

• ORB_ANONYMIZE=true (default): replaces repo/owner with a stable HMAC hash before sending
• ORB_INCLUDE_REVIEW_VERDICTS=false: disables the gate verdict join (for operators who want outcome data only)
• ORB_AIR_GAP=true: never sends externally; all data stays in the local DB for operator-only analysis

Central calibration service (future)

The orb.gittensory.app collector aggregates signals across all participating instances and publishes updated calibration parameters (gate thresholds, time-decay coefficients, AI prompt improvements) that self-hosted instances pull on a configurable schedule. This is the flywheel that makes the whole system self-improving over time.

Acceptance criteria

• docker compose --profile orb up starts Orb sidecar on port 8788
• Orb webhook validates App signature correctly (wrong secret → 401)
• pull_request.closed (merged) emits correct orb signal and posts to collector
• Revert detection fires on push with revert keyword matching a prior PR SHA
• ORB_ANONYMIZE=true replaces identifiers with HMAC hashes before sending
• ORB_AIR_GAP=true writes to DB only, never calls external URL
• All branches covered (merge/close/revert/air-gap/anonymize on/off)
• npm run test:ci green, Codecov ≥ 97% patch