Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ When you change runtime behavior, update the docs page **and** extend the drift

- **Core stack (default):** loopover + Redis + SQLite on the mounted data volume; `SELFHOST_DEPLOYMENT_MODE=dry-run` in `.env.selfhost.example`.
- **Optional profiles:** Postgres, REES sidecar, observability (Prometheus, Grafana, OTEL, Tempo, Loki), and backup — each documented on its concern page; none are required for a first healthy instance.
- **Sentry:** opt-in and off by default — set an operator-owned `SENTRY_DSN` or mount `SENTRY_DSN_FILE`; see [Operations](/docs/self-hosting-operations).
- **PostHog error tracking:** opt-in and off by default, replaces Sentry — set an operator-owned `POSTHOG_API_KEY`; see [Operations](/docs/self-hosting-operations).
- **AI / RAG / REES:** off in the starter env until webhook delivery and deterministic review are verified; each has explicit enablement switches.
- **Operator-owned paths:** `loopover-config/`, `loopover-data`, and secrets via `.env` or `*_FILE` mounts — never baked into images.

Expand Down
183 changes: 88 additions & 95 deletions apps/loopover-ui/content/docs/self-hosting-operations.mdx

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions apps/loopover-ui/src/lib/selfhost-docs-audit.ts
Original file line number Diff line number Diff line change
Expand Up @@ -179,11 +179,11 @@ export const SELFHOST_SOURCE_OF_TRUTH_ROWS: readonly SelfHostSourceOfTruthRow[]
driftGuard: "docs-selfhost-release-checklist-event-names.test.ts",
},
{
topic: "Sentry (opt-in, operator-owned DSN)",
runtimeSources: ["src/selfhost/sentry.ts", "docker-compose.yml"],
topic: "PostHog error tracking (opt-in, replaces Sentry)",
runtimeSources: ["src/selfhost/posthog.ts", "docker-compose.yml"],
docsPath: "/docs/self-hosting-operations",
driftGuard: "docs-selfhost-sentry-observability.test.ts",
notes: "Sentry is off by default until SENTRY_DSN or SENTRY_DSN_FILE is set.",
driftGuard: "docs-selfhost-posthog-observability.test.ts",
notes: "PostHog is off by default until POSTHOG_API_KEY is set.",
},
{
topic: "OTEL metrics/traces and Grafana dashboards",
Expand Down
43 changes: 14 additions & 29 deletions apps/loopover-ui/src/lib/selfhost-env-reference.ts
Original file line number Diff line number Diff line change
Expand Up @@ -429,13 +429,21 @@ export const SELFHOST_ENV_REFERENCE_ROWS: SelfHostEnvReferenceRow[] = [
name: "PORT",
firstReference: "src/server.ts",
},
{
name: "POSTHOG_API_KEY",
firstReference: "src/selfhost/otel.ts",
},
{
name: "POSTHOG_ENVIRONMENT",
firstReference: "src/server.ts",
firstReference: "src/selfhost/otel.ts",
},
{
name: "POSTHOG_HOST",
firstReference: "src/selfhost/otel.ts",
},
{
name: "POSTHOG_RELEASE",
firstReference: "src/selfhost/posthog.ts",
firstReference: "src/selfhost/otel.ts",
},
{
name: "PUBLIC_API_ORIGIN",
Expand Down Expand Up @@ -525,26 +533,6 @@ export const SELFHOST_ENV_REFERENCE_ROWS: SelfHostEnvReferenceRow[] = [
name: "SELFHOST_SETUP_TOKEN",
firstReference: "src/selfhost/preflight.ts",
},
{
name: "SENTRY_DSN",
firstReference: "src/selfhost/sentry.ts",
},
{
name: "SENTRY_ENVIRONMENT",
firstReference: "src/selfhost/otel.ts",
},
{
name: "SENTRY_RELEASE",
firstReference: "src/selfhost/otel.ts",
},
{
name: "SENTRY_SERVER_NAME",
firstReference: "src/selfhost/sentry.ts",
},
{
name: "SENTRY_TRACES_SAMPLE_RATE",
firstReference: "src/selfhost/sentry.ts",
},
{
name: "SETUP_OUTPUT_PATH",
firstReference: "src/server.ts",
Expand Down Expand Up @@ -664,8 +652,10 @@ export const SELFHOST_ENV_REFERENCE_MARKDOWN = [
"| `PGPOOL_MAX` | `src/selfhost/queue-common.ts` |",
"| `PGVECTOR_ENABLED` | `src/server.ts` |",
"| `PORT` | `src/server.ts` |",
"| `POSTHOG_ENVIRONMENT` | `src/server.ts` |",
"| `POSTHOG_RELEASE` | `src/selfhost/posthog.ts` |",
"| `POSTHOG_API_KEY` | `src/selfhost/otel.ts` |",
"| `POSTHOG_ENVIRONMENT` | `src/selfhost/otel.ts` |",
"| `POSTHOG_HOST` | `src/selfhost/otel.ts` |",
"| `POSTHOG_RELEASE` | `src/selfhost/otel.ts` |",
"| `PUBLIC_API_ORIGIN` | `src/selfhost/preflight.ts` |",
"| `PUBLIC_ORIGIN_ACKNOWLEDGED` | `src/server.ts` |",
"| `PUBLIC_SITE_ORIGIN` | `src/server.ts` |",
Expand All @@ -688,11 +678,6 @@ export const SELFHOST_ENV_REFERENCE_MARKDOWN = [
"| `SELFHOST_BUNDLE_ALL` | `scripts/build-selfhost.ts` |",
"| `SELFHOST_SERVICE` | `scripts/smoke-observability-traces.ts` |",
"| `SELFHOST_SETUP_TOKEN` | `src/selfhost/preflight.ts` |",
"| `SENTRY_DSN` | `src/selfhost/sentry.ts` |",
"| `SENTRY_ENVIRONMENT` | `src/selfhost/otel.ts` |",
"| `SENTRY_RELEASE` | `src/selfhost/otel.ts` |",
"| `SENTRY_SERVER_NAME` | `src/selfhost/sentry.ts` |",
"| `SENTRY_TRACES_SAMPLE_RATE` | `src/selfhost/sentry.ts` |",
"| `SETUP_OUTPUT_PATH` | `src/server.ts` |",
"| `SLACK_WEBHOOK_URL` | `src/services/notify-discord.ts` |",
].join("\n");
2 changes: 0 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 0 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -127,8 +127,6 @@
"@opentelemetry/sdk-trace-node": "^2.9.0",
"@sentry/cloudflare": "^10.63.0",
"@sentry/hono": "^10.63.0",
"@sentry/node": "^10.63.0",
"@sentry/opentelemetry": "^10.63.0",
"agents": "^0.17.3",
"aws4fetch": "^1.0.20",
"drizzle-orm": "^0.45.2",
Expand Down
1 change: 0 additions & 1 deletion scripts/branding-drift-baseline.json
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@
"src/selfhost/health.ts": 3,
"src/selfhost/monitored-work.ts": 1,
"src/selfhost/orb-collector.ts": 1,
"src/selfhost/sentry.ts": 2,
"src/services/ai-review.ts": 4,
"src/services/ai-slop.ts": 1,
"src/services/ai-summaries.ts": 1,
Expand Down
14 changes: 1 addition & 13 deletions src/env.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -281,17 +281,6 @@ declare global {
* Defaults to 60 when unset. This is on top of PagerDuty's own `dedup_key` coalescing (which prevents
* duplicate *incidents*, not duplicate *pages* for a still-open one). */
PAGERDUTY_COOLDOWN_MINUTES?: string;
/** Sentry noise control (#5119): the minimum severity (`info` < `warning` < `error` < `critical`) that
* reaches Sentry from captureError/captureReviewFailure/forwardStructuredLogToSentry (src/selfhost/sentry.ts),
* for any repo not present in SENTRY_REPO_MIN_SEVERITY (a JSON `{repoFullName: severity}` map, same
* deliberately-untyped pattern as PAGERDUTY_REPO_MIN_SEVERITY). Defaults to `error` when unset — matches
* every capture path's pre-#5119 behavior exactly, so an operator who never touches these vars sees no
* change. Lower a specific repo's threshold via the map (e.g. to `info`) for full visibility while
* actively debugging it, without raising Sentry noise everywhere else. Shares one severity-threshold
* resolver with PAGERDUTY_MIN_SEVERITY (src/services/severity-threshold.ts) — not a parallel concept. */
SENTRY_MIN_SEVERITY?: string;
/** Per-repo override map for SENTRY_MIN_SEVERITY — see its doc comment for the shape and precedence. */
SENTRY_REPO_MIN_SEVERITY?: string;
PRODUCT_USAGE_HASH_SALT?: string;
/** Server-to-server API bearer token — bypasses per-repo write checks (src/auth/security.ts). */
LOOPOVER_API_TOKEN?: string;
Expand Down Expand Up @@ -656,8 +645,7 @@ declare global {
* POSTHOG_API_KEY activates. Unset ⇒ the US-cloud default (https://us.i.posthog.com). Only meaningful
* alongside POSTHOG_API_KEY; ignored when unconfigured. Self-host error tracking also reads its own
* process.env-only vars (POSTHOG_MIN_SEVERITY, POSTHOG_REPO_MIN_SEVERITY, POSTHOG_ENVIRONMENT,
* POSTHOG_SERVER_NAME, POSTHOG_RELEASE) — not declared here, mirroring SENTRY_MIN_SEVERITY/
* SENTRY_REPO_MIN_SEVERITY's identical self-host-only precedent (src/selfhost/sentry.ts). */
* POSTHOG_SERVER_NAME, POSTHOG_RELEASE) — not declared here, self-host-exclusive config. */
POSTHOG_HOST?: string;
}
}
Expand Down
48 changes: 1 addition & 47 deletions src/queue/ai-review-orchestration.ts
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,6 @@ import {
resolveEnrichmentLinkedIssue,
resolveEnrichmentLinkedIssueNumbers,
} from "../review/enrichment-wire";
import { captureReviewFailure } from "../selfhost/sentry";
import { capturePostHogReviewFailure } from "../selfhost/posthog";
import { isReputationEnabled, shouldSkipAiForReputation } from "../review/reputation-wire";
import { isConvergenceRepoAllowed } from "../review/cutover-gate";
Expand Down Expand Up @@ -805,24 +804,8 @@ export async function runAiReviewForAdvisory(
action:
"The gate is held for a human reviewer rather than passed automatically; it re-evaluates on the next update.",
});
// A review that could not be produced is a real failure the maintainer must SEE — surface it to Sentry as an
// A review that could not be produced is a real failure the maintainer must SEE — surface it to PostHog as an
// ERROR (this also covers the INCOHERENT_DIFF bail, which parses to a missing opinion → inconclusive). (#1468)
captureReviewFailure(new Error("AI review inconclusive — no usable verdict for the PR head"), {
kind: "review",
reason: "ai_review_inconclusive",
installationId: args.installationId,
owner: args.repoFullName.split("/")[0],
repo: args.repoFullName,
pr: args.pr.number,
head_sha: args.advisory.headSha,
ai_review_mode: args.settings.aiReviewMode,
reviewer_count: result.reviewerCount,
public_notes: hasPublicReviewAssessment(result.advisoryNotes),
// Compact strings, not the raw objects -- Sentry's normalizeDepth flattens nested entries to "[Object]"
// and destroys the per-attempt detail (LOOPOVER-2B); see formatReviewDiagnosticsForCapture.
/* v8 ignore next -- current review runner always supplies diagnostics for completed AI attempts. */
review_diagnostics: formatReviewDiagnosticsForCapture(result.reviewDiagnostics ?? []),
}, "ai_review_inconclusive");
capturePostHogReviewFailure(new Error("AI review inconclusive — no usable verdict for the PR head"), {
kind: "review",
reason: "ai_review_inconclusive",
Expand Down Expand Up @@ -893,28 +876,6 @@ export async function runAiReviewForAdvisory(
};
findings.push(unavailableFinding);
args.advisory.findings.push(unavailableFinding);
captureReviewFailure(
new Error("AI review did not produce public notes for the PR head"),
{
kind: "review",
reason: "ai_review_public_summary_missing",
installationId: args.installationId,
owner: args.repoFullName.split("/")[0],
repo: args.repoFullName,
pr: args.pr.number,
head_sha: args.advisory.headSha,
ai_review_mode: args.settings.aiReviewMode,
reviewer_count: result.reviewerCount,
// Same "[Object]" flattening hazard as the inconclusive capture above (LOOPOVER-2B).
/* v8 ignore next -- current review runner always supplies diagnostics for completed AI attempts. */
review_diagnostics: formatReviewDiagnosticsForCapture(result.reviewDiagnostics ?? []),
configured_reviewers:
env.AI_REVIEW_PLAN?.reviewers?.map((reviewer) => reviewer.model) ??
null,
combine: env.AI_REVIEW_PLAN?.combine ?? null,
},
"ai_review_public_summary_missing",
);
capturePostHogReviewFailure(
new Error("AI review did not produce public notes for the PR head"),
{
Expand Down Expand Up @@ -958,13 +919,6 @@ export async function runAiReviewForAdvisory(
// error is a genuinely caught exception here (unlike the two captures above, which construct their own
// Error to report a known condition) -- named to mirror the structured log's own "event" field just above,
// not the exception's native class, so every unexpected review crash groups under one readable title.
captureReviewFailure(error, {
kind: "review",
installationId: args.installationId,
repo: args.repoFullName,
pr: args.pr.number,
head_sha: args.advisory.headSha,
}, "ai_review_failed");
capturePostHogReviewFailure(error, {
kind: "review",
installationId: args.installationId,
Expand Down
32 changes: 6 additions & 26 deletions src/queue/processors.ts
Original file line number Diff line number Diff line change
Expand Up @@ -584,7 +584,6 @@ import { commitE2eTestToPrBranch } from "../github/e2e-test-commit";
import { shouldApplyRepoCultureProfile } from "../review/repo-culture-profile-wire";
import { applyReviewMemorySuppression, getCachedReviewSuppressions, invalidateReviewSuppressionCache, shouldApplyReviewMemory } from "../review/review-memory-wire";
import { isEnrichmentEnabled } from "../review/enrichment-wire";
import { captureReviewFailure } from "../selfhost/sentry";
import { capturePostHogReviewFailure } from "../selfhost/posthog";
import {
setReviewPipelineSpanOutcome,
Expand Down Expand Up @@ -1119,7 +1118,7 @@ async function isRegateRepairExhausted(env: Env, repoFullName: string, pr: Pick<
});
// level:"error" is deliberate, not a code failure: this line only fires once the cap above already
// stopped the wasteful repair loop, so its OWN existence is the operator-visible signal (via the
// structured log → Sentry forwarder, forwardStructuredLogToSentry) that a PR kept failing repair for the
// structured log → PostHog forwarder, forwardStructuredLogToPostHog) that a PR kept failing repair for the
// same head SHA — the same "surface an anomaly at error level" convention selfhost_ai_provider_failed /
// selfhost_ai_providers_exhausted already use in src/selfhost/ai.ts.
console.error(
Expand Down Expand Up @@ -3904,15 +3903,15 @@ async function prReadyForReview(
}).catch(() => undefined);
// level:"error" is deliberate, not a code failure: this line only fires once the guard above already
// stopped the wasteful re-review, so its OWN existence is the operator-visible signal (via the structured
// log → Sentry forwarder, forwardStructuredLogToSentry) that a PR's CI has been permanently stuck long
// log → PostHog forwarder, forwardStructuredLogToPostHog) that a PR's CI has been permanently stuck long
// enough to need a human — the same "surface an anomaly at error level" convention selfhost_ai_provider_
// failed / selfhost_ai_providers_exhausted already use in src/selfhost/ai.ts. Rate-limited to once per
// (repo, pr, headSha) per day (#4998) — the defer above still runs on every evaluation; only the log is
// coalesced, so one permanently-stuck PR doesn't flood Sentry with hundreds of copies of the same signal.
// coalesced, so one permanently-stuck PR doesn't flood PostHog with hundreds of copies of the same signal.
if (!(await ciStuckRepeatLogCoalesced(env, repoFullName, pr.number, pr.headSha))) {
// LOOPOVER-2F: for a release-automation PR the "stuck" state is EXPECTED (see
// isReleaseAutomationHeadRef), so the once-a-day signal drops to warn — below the
// Sentry forwarder's error threshold, still in Workers Logs — while a real PR keeps
// PostHog forwarder's error threshold, still in Workers Logs — while a real PR keeps
// paging at error. Sink matches the stamped level per #7806 (console.warn for warn,
// console.error for error), which is why the whole call is picked, not just the level.
const releaseAutomation = isReleaseAutomationHeadRef(pr.headRef);
Expand Down Expand Up @@ -5707,7 +5706,7 @@ async function maybeHandleRepositoryRenamedWebhookEvent(
* (LOOPOVER_REPO_CONFIG_DIR/{owner}__{repo}/...), which the app can only READ (the mount is read-only)
* and which private-config.ts derives from the repo's CURRENT name. If one existed under the old name but
* not the new one, the operator's gate/autonomy/review policy for this repo just silently reverted to
* global defaults -- loud enough to reach Sentry (level:"error"), not a routine info line, because the
* global defaults -- loud enough to reach PostHog (level:"error"), not a routine info line, because the
* failure mode is exactly "reviews quietly stop matching what the operator configured," not a crash.
* A cloud deployment (no local reader registered) always resolves both sides false, so this never fires there.
*/
Expand Down Expand Up @@ -9159,16 +9158,7 @@ async function maybePublishPrPublicSurface(
},
});
// The advisory ran but NOTHING reached the PR (revoked token / perms removed / GitHub 5xx). For an
// advisory-only bot this is the worst failure — escalate to Sentry at error level, not just the audit ledger.
captureReviewFailure(new Error("PR public-surface publish failed — review produced output but nothing was posted to the PR"), {
kind: "publish",
installationId,
owner: repoFullName.split("/")[0],
repo: repoFullName,
pr: pr.number,
head_sha: advisory.headSha,
failedOutputs: failedOutputs.map((failure) => failure.output),
}, "pr_public_surface_publish_failed");
// advisory-only bot this is the worst failure — escalate to PostHog at error level, not just the audit ledger.
capturePostHogReviewFailure(new Error("PR public-surface publish failed — review produced output but nothing was posted to the PR"), {
kind: "publish",
installationId,
Expand Down Expand Up @@ -10367,16 +10357,6 @@ async function maybePublishPrPublicSurface(
aiReviewMode: settings.aiReviewMode,
},
}).catch(() => undefined);
captureReviewFailure(new Error(message), {
kind: "review",
reason: "ai_review_public_summary_missing",
installationId,
repo: repoFullName,
pr: pr.number,
head_sha: advisory.headSha,
reviewer_count: aiReview?.reviewerCount ?? 0,
public_notes: hasPublicReviewAssessment(aiReview?.notes),
}, "ai_review_public_summary_missing");
capturePostHogReviewFailure(new Error(message), {
kind: "review",
reason: "ai_review_public_summary_missing",
Expand Down
4 changes: 2 additions & 2 deletions src/review/active-review-reconciliation.ts
Original file line number Diff line number Diff line change
Expand Up @@ -115,8 +115,8 @@ export async function runActiveReviewReconciliation(env: Env, nowMs: number = Da
// warn, not error (LOOPOVER-2K): a successful terminalization is this feature WORKING, not an anomaly --
// logging it at error level turned a backlog drain into 547 Sentry error events in a day (one per healed
// row, message-fingerprint-collapsed into a single escalating issue). warn stays visible in stdout/
// Workers Logs but sits below the default SENTRY_MIN_SEVERITY of "error" (selfhost/sentry.ts's
// resolveSentryMinSeverity), so Sentry only sees it when an operator has explicitly lowered the
// Workers Logs but sits below the default POSTHOG_MIN_SEVERITY of "error" (selfhost/posthog.ts's
// resolvePostHogMinSeverity), so PostHog only sees it when an operator has explicitly lowered the
// threshold. The row_error/scan-error paths below stay at error: a FAILED heal is still an anomaly.
console.warn(
JSON.stringify({
Expand Down
2 changes: 1 addition & 1 deletion src/selfhost/host-pressure.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
// Optional host-CPU-pressure hint for maintenance-job admission (see maintenance-admission.ts). Node-only --
// `node:os`'s loadavg() has no meaningful signal on Cloudflare Workers -- this module is imported ONLY by the
// self-host Node queue backends (sqlite-queue.ts / pg-queue.ts), never by src/index.ts's Worker bundle, so a
// static `node:os` import here is safe (mirrors the existing `hostname` import in selfhost/sentry.ts).
// static `node:os` import here is safe (mirrors the existing `hostname` import in selfhost/posthog.ts).
import { cpus, loadavg } from "node:os";

/** The 1-minute load average normalized per logical core, so the SAME threshold means the same thing on a
Expand Down
Loading
Loading