feat(pricing): add OpenRouter as second fallback pricing source - #3
Closed
godlockin wants to merge 169 commits into
Closed
feat(pricing): add OpenRouter as second fallback pricing source#3godlockin wants to merge 169 commits into
godlockin wants to merge 169 commits into
Conversation
…n-io#975) The single-session freshness fast path skips re-parsing a Claude JSONL source when its size, mtime, and file identity (inode/device) all match the stored row. A same-size, same-mtime, same-inode **in-place** rewrite matches all three signals, so the engine keeps the stale rows instead of re-parsing. This happens in the wild when two writes land in one filesystem mtime granule, or on a coarse-mtime filesystem. The inode net (kenn-io#357) only catches atomic-rename replacements (the inode changes); it is blind to in-place rewrites. The same gap produced a nondeterministic CI flake in `TestIncrementalSync_ClaudeSameSizeFileReplaceUsesFullParse` on filesystems where the inode signal is unavailable and the check degrades to mtime alone. Fix: a content-hash tie-breaker that runs **only** once size + mtime + identity already match — it compares the on-disk prefix hash (over the stored `file_size`, which `shouldSkipFile` has confirmed equals the current size) against the stored `file_hash`, and forces a full parse on mismatch. The common path stays cheap: a real append changes the size and returns before hashing, so the hash is paid only for sources that already look byte-for-byte unchanged — the same skip-path-hashing the codebase already uses for coarse-mtime sources (S3/Aider/Shelley). Rows without a stored hash fall back to the prior size/mtime/identity behavior. Claude's incremental path now refreshes `file_hash` (previously only Codex did), so the stored hash stays a valid current fingerprint. Covers local **and** remote (path-rewritten) sync: for remote sources the physical materialized file is hashed (the logical rewritten key is not openable), and the stored hash is computed over those same materialized bytes on both the full-parse and incremental paths, so the comparison is directly comparable — an unchanged re-download still hashes equal and skips, while a genuine rewrite re-parses. Tradeoff: the content hash is a recurring read on the freshness fast path for otherwise-unchanged Claude sessions, bounded by the sync cutoff window. It is the deliberate cost of closing a correctness gap (silently serving stale content) that stat signals cannot detect. Where to look: `internal/sync/engine.go` — `providerIncrementalContentChanged` and the `statPath` capture in `providerSingleSessionFresh`, plus the Claude `file_hash` refresh in `tryIncrementalJSONL`/`writeIncremental`. Tests in `internal/sync/engine_integration_test.go` cover the local in-place rewrite, the remote path-rewritten rewrite, and the unchanged-remote-re-download skip. Co-authored-by: Matthew Jacobs <mjacobs@users.noreply.github.com>
…unter (kenn-io#969) Extends the per-agent parser-anomaly counters (kenn-io#866, kenn-io#948) with a new signal: Antigravity sessions that carried `gen_metadata` rows but decoded into zero usage events. That combination is an early warning that a newer Antigravity build changed the gen_metadata token-block wire format the decode heuristic depends on — today it fails silently, and the session simply shows no usage. The signal is set by both Antigravity parsers (IDE and CLI) as a transient flag on `ParsedSession`, computed from the *final* usageEvents so a CLI session whose gen_metadata failed to decode but whose trajectory sidecar supplied usage is correctly not flagged. It is recorded per agent at the existing `prepareSessionWrite` seam and rendered in the CLI sync summary next to the malformed-lines and unrecognized-schema counters, mirroring the kenn-io#953 `UnknownSchemaSessionsByAgent` counter. Deliberately schema-free: the flag never becomes a persisted column, so there is no migration and nothing flows to PostgreSQL/DuckDB — `AnomalyStats` is consumed only by the CLI summary printers. Where to look: `internal/parser/antigravity.go` and `antigravity_cli.go` (detection and the sidecar-rescue correctness point), `internal/sync/progress.go` (`AnomalyStats`), `internal/sync/engine.go` (recording seam), `cmd/agentsview/main.go` (render). Co-authored-by: Matthew Jacobs <mjacobs@users.noreply.github.com>
…#980) Fixes kenn-io#959 ## What OMP (Oh My Pi) v16.3+ writes a fixed-width, rewritable `{"type":"title",...}` slot as the first line of each session file — exactly 256 UTF-8 bytes including newline, space-padded via its `pad` field — so the current title can be updated in place without shifting the rest of the file ([`session-title-slot.ts`](https://github.com/can1357/oh-my-pi/blob/master/packages/coding-agent/src/session/session-title-slot.ts)). The `{"type":"session",...}` header now sits on the second line. `IsPiSessionFile` sniffed only the first non-blank line for a session header, so discovery rejected every OMP session. `parsePiLikeSession` made the same first-line assumption one layer deeper and would have refused to parse them regardless. ## Changes - `IsPiSessionFile` skips `"title"`-typed lines before deciding, staying in sync with the parser's header scan (the two are documented to match). - `parsePiLikeSession` skips the slot in its header scan and keeps its title. Session name precedence: slot title (rewritten in place, always current) > `session_info` renames (pi lineage, which has no slot) > v3 header `title` (the initial auto-generated title). - Plain pi files without the slot behave exactly as before, and a title-slot-only file with no session header is still rejected. ## Where to look - `internal/parser/discovery.go` — the sniff loop - `internal/parser/pi.go` — header scan and name precedence ## Limitations / follow-ups - OMP records branch lineage as `parentSession` (a session ID) where pi uses `branchedFrom` (a file path), so `ParentSessionID` stays empty for branched OMP sessions; left for a follow-up. - Other OMP-specific entry types (`title_change`, `custom`, `custom_message`, the `fileMention` message role) continue to fall through the existing skip-silently paths. Co-authored-by: Matthew Jacobs <mjacobs@users.noreply.github.com>
…sync baseline (kenn-io#970) parse-diff v2 follow-up (after kenn-io#805 sub-items 1+4 and kenn-io#949 sub-item 3), addressing incremental-append skew. `parse-diff` compares stored session rows against a fresh full re-parse. Sessions last written through the incremental JSONL append path (Claude/Codex) are not rewritten through full normalization, so a fresh full re-parse can legitimately differ on the per-message metadata / ordinal-set shape even at the current data_version and without a pending resync. Those sessions were reported as changed — noise that makes `--fail-on-change` untrustworthy in CI against a non-resynced archive. This adds a SQLite-only `sessions.last_write_incremental` marker, set inside the single incremental-update seam. It is reset to false only by a genuine full message re-normalization (`ReplaceSessionContent`, `ReplaceSessionMessages`, and the batch `ReplaceMessages` branch) via a shared `resetIncrementalMarkerTx`, and seeded false on fresh INSERT — not on a bare `UpsertSession`. That distinction matters: the routine full-parse batch path takes `ReplaceMessages=false` for the append-only agents, upserting the session row while appending only new messages and leaving earlier incrementally written rows in place, so clearing the marker on every upsert would have made a single routine sync report still-present benign skew as real drift. The marker is therefore per-session ground truth (not inferred) and self-heals on the next full resync. A new `DiffIncrementalSkew` class — mirroring `DiffRaced` — reclassifies a would-be change on such a row: it is listed for drill-down but excluded from failure counting, while still counting toward Examined. Precedence is raced > incremental-skew > changed. The CLI additionally prints a resync-baseline recommendation whenever skew is present. The reclassification is confined to the incremental-artifact surface, not applied session-wide. Because the marker is session-level while a regression is field-level, the marker only reclassifies when every non-informational diff is on an allow-list holding just the ordinal-shape / per-message metadata field (`message_metadata`, via `diffsConfinedToIncrementalArtifacts`). A non-informational diff on any other field — head-derived `first_message`/`started_at`, message content, usage totals, tool_calls, or the recomputed aggregates, all of which both write paths normalize identically — keeps the session `DiffChanged` and still trips `--fail-on-change`. The session fields the incremental path actually freezes (`termination_status`, `session_name`, `cwd`, …) are already marked informational and never reach this test. Characterized against the real archive: `parse-diff` over the full Claude/Codex corpus shows 0 changed and 0 skew for quiescent current-version rows; the only per-message metadata drift observed was on a live-write raced session, which already outranks skew. The class is thus a conservative safety net for the one surface a whole-file re-parse can legitimately reshape, without masking regressions elsewhere. Backend note: the marker is SQLite-only by design, like `next_ordinal`/`last_entry_uuid`/`file_hash`. parse-diff opens the local SQLite archive directly and is never reachable through the PostgreSQL/DuckDB read paths; the column is `json:"-"` and the push paths use explicit column lists that already omit the sync-bookkeeping cluster. The migration is a non-destructive ADD COLUMN with no dataVersion bump. Where to look: `internal/db/sessions.go` / `messages.go` / `session_batch.go` (set, scan, and the full-rewrite-only reset), `internal/sync/parsediff_compare.go` and `parsediff.go` (classify, compute, and artifact confinement), `internal/sync/parsediff_report.go` (the class), `cmd/agentsview/parse_diff.go` (render + resync note). Co-authored-by: Matthew Jacobs <mjacobs@users.noreply.github.com>
Filtered PostgreSQL pushes can still fall back to repeated full sweeps after the project filter set changes, because the one-time session-alias backfill marker is stored in the same project-filtered sync-state scope as the incremental push watermarks. The filtered scope is correct for `last_push_at`, target fingerprints, and boundary fingerprints, but it is the wrong scope for a migration marker that only needs to complete once per PG target. This moves the alias-backfill marker onto target-scoped sync state while leaving the ordinary filtered push state in the existing project-filtered store. Changing `--projects` or `--exclude-projects` will no longer re-arm the alias-backfill full-push gate after the same target has already completed it, but different filter sets will still keep their own incremental watermarks and reset behavior exactly as before. The change stays entirely inside `internal/postgres`, and it keeps the current one-time marker model rather than widening the fix into per-session tracking. One bounded limitation remains: the first filtered push that sets the target-scoped marker only backfills sessions inside that filter. Sessions outside that filtered run keep their old alias state until a later push includes them or they change locally, so this narrows the repeated full-push bug without claiming full per-session coverage. This follows the narrowed analysis in [kenn-io#939](kenn-io#939) from leejuhanKr and the split left by merged sibling [kenn-io#940](kenn-io#940). Closes kenn-io#939 Co-authored-by: Rod Boev <rodboev@users.noreply.github.com>
## Add Devin CLI as a supported local session source. This teaches AgentsView to discover Devin roots from the local-share directory, parse session metadata from cli/sessions.db, read transcript content from cli/transcripts, and surface Devin sessions in the UI and settings docs. Devin is integrated through the provider facade because its session identity, freshness, and watch behavior come from a composite DB-plus-transcript source rather than a single transcript file. The parser now handles Devin transcript messages, tool calls/results, token usage, privacy-safe error reporting, and fallback parsing from message_nodes when transcript JSON is missing. Sync and watcher integration were updated so Devin sessions use provider-owned virtual source paths, composite freshness, and provider watch plans without widening remote sync or other non-Devin behavior. Reviewers should focus on: - internal/parser/devin.go - internal/parser/devin_provider.go - internal/sync/engine.go - cmd/agentsview/main.go We intentionally narrow earlier provider-wide fallout so SSH remote sync, parse-diff, token-use probing, and resync safety counting stay scoped to existing behavior, with Devin as the only new non-file-backed exception. Co-authored-by: Aaron Florey <aaronflorey@users.noreply.github.com>
…#957) Migrates the frontend to the shared `@kenn-io/kit-ui` component library in the six staged commits from kit-ui's migration guide, each independently reviewable and validated: design tokens, display primitives (Spinner, StatusDot, KbdBadge, EmptyState), stateful primitives (CopyButton, Button, TableHeaderCell, Tooltip), overlays and layout (Modal, Typeahead, FilterDropdown, FindBar, StatusBar, TopBar, DateRangePicker, RefreshControl), utilities and the theme store, and finally enforcement. **Dependency strategy.** kit-ui is consumed as a commit-pinned GitHub git dependency (`github:kenn-io/kit-ui#215d252` in `frontend/package.json`). The repo is public, so `npm ci` clones it anonymously over HTTPS everywhere — no sibling checkout, no registry publish, no CI credentials. Bumping the dependency is a one-line hash change plus `npm install`. The Docker build needs no special wiring. **What stays local, and why.** Thin wrappers inject what kit-ui can't know: localized strings and the app locale (`shared/RangePicker.svelte`, `shared/RefreshControl.svelte`), store wiring (`SessionFindBar`), and domain glue (`ProjectTypeahead`, the modal family). The i18n-aware formatters, keyboard shortcuts, markdown pipeline, and `content/CodeBlock` (controlled-copy contract) intentionally remain app-owned; DESIGN.md documents each boundary plus the two remaining upstream gaps (Modal close-X aria-label, TopBar no-active-tab state). **Deliberate behavior changes.** - Relative date windows now follow kit-ui's semantics: "Last N days" spans N calendar days inclusive of today (previously N+1). Stage 4 had left the picker displaying kit-ui's ranges while queries resolved the old ones; every consumer (preset resolution, rolling `window_days` URLs, custom-tab seeding, analytics/usage stores) now agrees. Pinned from/to URLs are unaffected. - The theme store moved to kit-ui: existing `"theme"` localStorage values carry over, the legacy high-contrast key migrates at startup, and users without a stored preference gain OS-preference tracking. - Responsive breakpoints snapped to the shared 640/760/900 ladder, with JS layout logic (`ui.isMobileViewport`, `SIDEBAR_DESKTOP_BREAKPOINT`) derived from the same constants; the insights generated-archive grids collapse on container width because their hard column minimums make viewport gates wrong when the sidebar is open. **Enforcement.** `npm run check:kit-ui` gates CI at zero findings (raw colors tokenized, spacing on the `--space-N` ladder, shared breakpoints, `kit-popover-card` chrome). The gate is the plain full-rule `kit-ui-check src` run — no rules disabled, no paths exempted. Every legitimate exception carries an in-file `kit-ui-check-ignore` marker with a reason: the definitional token lines in `app.css`, the trends brown palette slot, and CodeBlock's copy contract. **Where to look.** The riskiest changes are the date-window semantics (`utils/dates.ts`, `shared/dateRangeSelector.ts`, and the store updates in `stores/analytics.svelte.ts`/`stores/usage.svelte.ts`) and the theme-store delegation in `stores/ui.svelte.ts`. The stage-6 commit is broad but style-only. Visual deltas from token snapping were kept conservative but were not screenshot-verified; the deliberate ones are documented in the stage commit messages. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <sup>generated by a clanker</sup> Co-authored-by: Marius van Niekerk <mariusvniekerk@users.noreply.github.com>
… config and errors (kenn-io#968) Hardening for the sync path, from daemon startup through full resync to HTTP remote sync. The work started with a resync that took 6m45s (vs a 4m30s baseline) followed by minutes of ~100% CPU, and grew to cover the failure modes hit while diagnosing it: a resync that silently wiped usage costs, a remote daemon that was unreachable with no indication why, and signal/findings writes that could go stale. ## serve status startup transparency While the daemon starts, `serve status` printed only `agentsview is starting up.` — no pid, elapsed time, or progress, even though a full resync can run for minutes and a background launch hides all output in serve.log. The starting daemon now publishes `startup-state.json` in the data dir (pid, started_at, phase, throttled sync progress detail, and the serve.log path for background children). `serve status` renders it: ``` agentsview is starting up. pid: 48151 elapsed: 1m12s phase: full resync: claude: 12340/38209 sessions (32%) log: ~/.agentsview/serve.log ``` The file is written atomically, updated at phase transitions and at most once per second from sync progress callbacks, and removed together with the start lock — readers only trust it while `IsDaemonStarting` holds, so staleness needs no handling. Missing or corrupt state falls back to the bare message; write failures never interrupt startup. ## Skip resync copy sanitize for already-sanitized sources The copy-time sanitize pass added at dataVersion 58 (kenn-io#945) streams every preserved message, tool call, and tool result event through Go `SanitizeUTF8` inside the copy transaction. On an archive with 83k orphaned sessions this took 2m40s per resync (measured against a 1m1s pre-kenn-io#945 baseline for the same phase), while only ~7.4k fields out of millions needed fixing. The gate is two-tier because ingest coverage grew in two steps: message content, tool results, and result events skip for sources at v58+, while `tool_calls.input_json` — which ingest did not sanitize until this PR — skips only at v59+. `dataVersion` is bumped to 59 so existing archives full-resync once: live sessions re-ingest through the extended `SanitizeMessage`, preserved orphaned/trashed rows get the copy-time input pass, and the 58-to-59 upgrade pays a single-column scan instead of the full pass. The threshold constants document that they must be bumped if `SanitizeUTF8` ever gains rules that need to apply to stored rows. Measured effect: the copy phase dropped from 2m39s back to 1m11s on the same archive. ## Filter signals backfill by stored version without marker `BackfillSignals` dropped the `quality_signal_version` filter whenever the completion marker was unset, recomputing every session in the archive. Post-resync databases always lose the marker when orphans are copied, so each data-version bump burned ~100% CPU re-deriving signals that were already current (113,923 sessions on a real archive, all at the current version, confirmed via `serve --pprof`). The filter now always applies — the column defaults to 0, so never-computed sessions stay eligible — and partial-run retries resume with only the sessions that still need work. To keep the filter sound, the version column is now a truthful invariant: secret findings persist before the version-advancing signals write (both orders of the recompute path), and message-appending transactions (`InsertMessages`, `WriteSessionIncremental`, `ReplaceSessionMessages`) zero `quality_signal_version` in the same transaction so appended content can't hide behind a current version. A `doctor sync` check counts sessions whose signals are current but have no persisted secret scan and suggests `agentsview secrets scan`. ## Preserve model_pricing across the resync swap A resync builds a fresh database and carried only insights, sync state, and archived sessions across the swap — `model_pricing` was dropped. The startup pricing seed runs once per daemon lifetime, so a resync triggered through the sync API (`agentsview sync --full` against a running daemon) left every usage cost reading $0.00 until the next restart. The swap now copies `model_pricing` (including its sentinel meta rows) alongside insights; a copy failure logs and warns rather than aborting the resync, since the next startup seed heals it. ## Persistent bind host for remote sync targets HTTP remote sync targets must listen on a non-loopback interface, but the bind address was flag-only: a remote daemon restarted without `--host` silently rebound to 127.0.0.1 and every sync failed. `host` is now a supported config.toml key so the bind survives restarts and auto-started daemons. A non-loopback config-file host requires `require_auth = true` — the server refuses to start rather than persistently exposing an unauthenticated API — while the `--host` flag keeps its existing per-invocation behavior and still overrides the file. ## Actionable HTTP remote sync errors Every HTTP remote sync failure collapsed to the string `HTTP remote sync failed`, hiding whether the daemon was unreachable, the token mismatched, or the remote was outdated. `remotesync` now returns a typed `StatusError` for non-2xx responses and `FailureSummary` maps errors to sanitized, actionable messages (connection refused → check the remote bind; 401/403 → token must match the remote `auth_token`; 404 → remote daemon predates the sync endpoints; DNS and timeout cases). Unknown errors still collapse to the generic message so raw URLs and response bodies never reach the API response; the server logs the raw error locally. ## Where to look - `cmd/agentsview/startup_state.go` — new state file writer/reader; lifecycle tied to the daemon start lock in `daemon_runtime.go`. - `internal/db/orphaned.go` — `sourceContentSanitized` gate and threshold rationale. - `internal/db/signals.go` — the backfill candidate query and its accepted edge case. - `internal/db/pricing.go` / `internal/sync/engine.go` — `CopyModelPricingFrom` and its call in the swap sequence. - `internal/config/config.go` / `cmd/agentsview/managed_caddy.go` — the `host` key and the non-loopback auth guardrail. - `internal/remotesync/failure.go` — the failure classifier. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Wes McKinney <wesm@users.noreply.github.com>
The 0.36.1 release includes several user-visible sync, parser, remote sync, and UI consistency changes, but the public docs still pointed at 0.36.0 as the latest release and did not explain the new diagnostics or troubleshooting surface. This adds release notes with contributor acknowledgements and updates the relevant reference pages so operators can interpret parse-diff incremental skew, startup/resync progress, Antigravity anomaly summaries, and HTTP remote sync failures without reading implementation PRs.
…io#977) (kenn-io#987) `frontend/src/lib/api/generated/services/AnalyticsService.test.ts` is a handwritten Vite test inside the generated client output tree. That placement conflicts with the generated-client contract in `docs/internal/huma-api-routes.md`: regeneration owns `frontend/src/lib/api/generated`, and the whole subtree is marked generated for GitHub. This moves the test up to `frontend/src/lib/api/AnalyticsService.generated-client.test.ts`, keeping the existing request assertion intact while updating the relative import and mocks. The generated client stays committed and unchanged; this slice only makes the generated tree contain generated output again so the follow-up freshness gate can run without deleting handwritten tests. Focused validation covers the relocated Vite test, frontend typecheck, the absence of tracked `.test.ts` files under `frontend/src/lib/api/generated`, and the relocated file's generated-file attribute. Refs kenn-io#977 Co-authored-by: Rod Boev <rodboev@users.noreply.github.com>
… to Claude Code (kenn-io#981) ## What Adds a `testing-without-tautologies` agent skill under `.agents/skills/`, ported from the middleman repo and re-grounded in this codebase, and wires up `.claude/skills/` so project skills are actually discoverable by Claude Code. ## Why Agent-written tests routinely pass without being able to fail: mirror assertions that compute the expectation with the code under test, mocks that accept anything, tests that re-prove `net/http` or FTS5 instead of our boundary contract, and — during cleanups — guard tests asserting a deleted function or file stays deleted. The skill makes the repo's testing expectations enforceable at authoring time instead of at review time, framed around one question: what production change should make this test fail? Separately, skills here canonically live in `.agents/skills/`, but Claude Code only discovers project skills under `.claude/skills/` — so the existing `localization-paraglide` skill was invisible to Claude Code sessions. Tracked relative symlinks (the same pattern middleman uses) let both harness layouts read one skill source. ## What's agentsview-specific - Examples target this repo's surfaces: hand-written parser JSONL fixtures (never generate expectations by running the parser), `testDB(t)`/`httptest`/`t.TempDir()`, SSE payloads, FTS query construction. - A Backend Parity section: a contract protected by an `internal/db` test must also be protected by the `pgtest`-tagged `internal/postgres` test, since a parity bug only one backend's suite can catch will be missed. - Codifies existing prose rules with teeth: the testify `require`/`assert` split, shell-script tests must exercise behavior rather than grep the script source, and no fabricated DB state to provoke errors (test the error-to-message mapping via a typed error instead). - A new rule with no middleman counterpart: never write negative-existence tests. Deletion is proven by the deletion, the build, and the replacement's behavior tests; a "still-deleted" assertion protects nothing, blocks future reuse of the name, and outlives the migration it policed. ## Limitations / follow-ups - New skills need both the `.agents/skills/` entry and a matching `.claude/skills/` symlink; nothing enforces that pairing yet. - Symlinks require a checkout that preserves them (fine on macOS/Linux; Windows checkouts need `core.symlinks=true`). ## Where to look - `.agents/skills/testing-without-tautologies/SKILL.md` — the skill itself; the Required Checks and Backend Parity sections carry the substance. - `.claude/skills/` — the two symlinks. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Marius van Niekerk <mariusvniekerk@users.noreply.github.com>
…kenn-io#985) ## Why agentsview has repeatedly shipped the same performance regression shape: sync work that should scale with new data silently starts scaling with archive size, and nothing in CI notices until a machine spends minutes in a sync pass. Recent history: discovery recomputing root-derived project info per source (kenn-io#912), the provider migration dropping pre-parse freshness skips, O(session-history) work on every streamed append (kenn-io#954), bulk-ingest throughput (kenn-io#411), and per-row `json_extract` in the usage scan (kenn-io#309). Each fix landed without a gate, so any refactor could quietly reintroduce its class. ## What this adds **Layer 1 — deterministic work-count invariants** in the normal test suite (immune to runner noise): `TestWarmFullSyncDoesNoBulkWriteWork` asserts a second full sync over an unchanged Claude archive skips every session and runs zero bulk-write batches, via the existing `SyncStats`/`PhaseStats` counters. It complements the existing Vibe freshness, signal-scheduler debounce, and parser once-per-root seam tests; `docs/internal/performance-gates.md` maps each historical regression class to the gate that covers it. **Layer 2 — a per-PR benchmark gate** (`.github/workflows/bench.yml`): hot-path benchmarks (warm no-op full sync, incremental append into a 1k-message session, cold-archive ingest through both the default and the resync bulk-write pipelines, streaming chunk-merge replace, batched insert, plus the existing usage and secret-scan benchmarks) run on the PR head and its merge base in the same job on the same runner, each side via `make bench-gate` — the Makefile is the single source of truth for the gated package list, sample count, and iteration count — and are compared by `cmd/benchgate`. `benchgate` builds on `golang.org/x/perf` (`benchfmt` parsing, `benchmath` — the statistics behind benchstat) and adds only the policy benchstat doesn't provide: thresholds, floors, and a failing exit code. The policy is tuned for shared-runner noise: - Gating is per benchmark; nothing is averaged across benchmarks. - `allocs/op` (1.25x) and `B/op` (1.35x) are deterministic for fixed code and iteration count, so they gate the candidate's **worst** `-count` run against the baseline median — a single outlier run is a real intermittent allocation path and fails. Failure lines include the baseline's worst run so pre-existing instability is visible. - Time gates medians at a loose 2.0x **and** requires Mann-Whitney significance, so one slow run can't flake a PR but algorithmic blowups fail. Fewer than 5 candidate samples is a loud configuration error; a baseline with fewer than 5 samples (a legitimately partial base run) is reported and not gated. - Benchmarks present on only one side never gate, which makes the gate allowlist-free: every benchmark in the gated packages runs, and new ones auto-join once they exist on both sides. Because each side runs its own commit's `make bench-gate`, a PR that grows `BENCH_GATE_PACKAGES` cannot break the base run. - A failing merge-base run degrades to whatever partial output it produced, plus a workflow warning — one broken package does not erase the baseline for the others. - Captures with unparseable result lines (for example log output interleaved into a `Benchmark` line) fail loudly with exit 2 instead of silently dropping those benchmarks from both sides; the sync benchmarks silence the engine's logger for exactly this reason. A gated unit the baseline has but the candidate lost (e.g. `-benchmem` dropped) is also a configuration error; a unit missing from the baseline, and custom `b.ReportMetric` units, are reported as not gated. - The gate runs a fixed `-benchtime=20x` so baseline and candidate measure identical workloads (two benchmarks deliberately grow their fixture per iteration); the count and benchtime are evaluated from the PR head's Makefile and passed into the merge-base run so changing them cannot skew the comparison. Benchmarks keep fixture construction and one-time leading-edge work out of the timed region so the gated ratios measure product cost, not test-helper cost. ## Tradeoffs and limitations - Subtle wall-clock regressions (< 2x or statistically ambiguous) are deliberately not time-gated; the allocation metrics act as the sensitive proxy for O(archive)-vs-O(delta) mistakes. - On this PR itself, the merge base predates the `bench-gate` Make target, so the baseline step degrades to empty (with a visible warning) and every benchmark reports as new; the gate is fully live from the first PR after this merges. - The resync sanitize-pass class (fixed on the unmerged `heavy-angora` branch) is not gated yet; a gate would fail against current main until that lands. - `golang.org/x/perf`'s module requirements force minor bumps of `x/net`, `x/oauth2`, `x/crypto`, `x/exp`, and `x/text` via MVS. - The workflow runs only on PRs that touch Go files, `go.mod`/`go.sum`, the Makefile, or the workflow itself; docs- and frontend-only PRs skip it. Cross-backend query benchmarks stay in the opt-in `internal/backendbench` (Docker) and are not part of the gate. ## Where to look - `cmd/benchgate/` — the comparison policy (worst-run gating, significance requirement, corrupted-capture and missing-unit reporting) - `.github/workflows/bench.yml` — base-vs-head orchestration and partial-baseline degradation - `internal/sync/engine_bench_test.go`, `internal/db/messages_bench_test.go`, `internal/sync/perf_invariant_test.go` - `docs/internal/performance-gates.md` — regression history and how to add a gated benchmark 🤖 Generated with [Claude Code](https://claude.com/claude-code) <sup>generated by a clanker</sup> Co-authored-by: Marius van Niekerk <mariusvniekerk@users.noreply.github.com>
…o#990) ToolBlock currently shows raw tool input and result text without the copy affordance used by adjacent content surfaces. Task prompts, fallback command input, and tool result output all render as escaped pre text in the session view, but copying them requires manual selection and long fallback input only shows the 20-line preview until expanded. This adds ToolBlock-local copy controls for the raw input or prompt source and `result_content`, reusing the existing `CopyButton` and app clipboard helper pattern from code blocks, message headers, and grouped tool calls. The input copy source stays separate from the displayed preview so long Bash fallback content copies the full raw command before "show all" is clicked, and the output copy control copies the exact result string without toggling the output section. The scope is deliberately presentation-only: ToolBlock, its focused tests, and synchronized locale labels. It does not add raw/formatted rendering toggles, path display settings, persistent preferences, parser behavior, backend behavior, or generalized copy handling outside ToolBlock. Validation is focused on ToolBlock copy behavior, locale compilation, and the frontend typecheck; CI covers the broader matrix. Refs kenn-io#984 Co-authored-by: Rod Boev <rodboev@users.noreply.github.com>
DuckDB mirror pushes now avoid rewriting session dependents when the target already matches local message, tool, result-event, and usage fingerprints. Suffix-only active-session growth appends just the new messages and dependent rows, while historical or dependent-row mismatches still fall back to a full replacement. The skip and append fast paths refresh per-session pinned messages and secret findings in the same mutation transaction, so a later partial push failure cannot leave those dependent rows stale while persisting fresh fingerprints. Incremental DuckDB pushes also repair cached fingerprints whose mirror rows are missing by re-adding the unchanged local session to the push candidates before skip filtering. That keeps retry and partial-failure cleanup from advancing the watermark while the mirror is still missing a session. The DuckDB backend runs threshold-gated checkpoint maintenance after committed mutating pushes, before local watermark advancement. PostgreSQL skip fingerprints now include tool-result events so both push backends agree on stale event detection. The main review points are `internal/duckdb/sync.go`, `internal/duckdb/push.go`, `internal/duckdb/push_fingerprint.go`, `internal/duckdb/checkpoint.go`, and the lifecycle coverage in `internal/duckdb/sync_fastpath_test.go`. Checkpointing is intentionally partial reclamation; large historical rewrites may still require full-copy compaction outside the normal push path. Co-authored-by: Phillip Cloud <cpcloud@users.noreply.github.com>
…#995) OMP (Oh My Pi) v3 session headers record branch lineage as `parentSession` — the parent's session ID — while upstream pi records `branchedFrom`, a file path. `parsePiLikeSession` only read `branchedFrom`, so branched OMP sessions never got a `ParentSessionID` and their lineage never resolved. Follow-up to kenn-io#980, which made these sessions discoverable in the first place. This maps `header.parentSession` to `ParentSessionID` as an OMP-only fallback when `branchedFrom` is absent, reusing the session's own ID prefix. Because a stored session ID is the ID prefix plus the header's session id, and `parentSession` carries the parent's raw session ID, the mapped value equals the parent's stored ID and lineage resolves with no extra lookup. `branchedFrom` still wins when present, and pi sessions never consult `parentSession`, so upstream pi behavior is unchanged. Where to look: the branch-lineage block in `internal/parser/pi.go`; `TestPiProviderOMPParentSessionMatchesParentID` pins that a branched child's mapped `ParentSessionID` equals its parent's stored ID. Limitation: other OMP-specific entry types observed in the same investigation (`title_change`, `mcp_tool_selection`, `custom`, `custom_message`, `fileMention`) still fall through silently; they carry no lineage information and are deliberately left as-is. Co-authored-by: Matthew Jacobs <mjacobs@users.noreply.github.com>
…n-io#996) Updates the frontend JavaScript dependency set, including the @kenn-io/kit-ui digest bump, Playwright, TanStack Virtual, testing-library/svelte, shiki, svelte, and svelte-check. The kit-ui bump also brings stricter conformance checks, so this branch adopts shared z-index tokens, SearchInput for Recent Edits, the shared sr-only class, and direct kit-ui debounce imports. The remaining app-owned undo toast, markdown renderer, and message virtualizer keep their current behavior and carry documented kit-ui-check exceptions because replacing them would be separate migrations. The docs screenshot package gets the matching Playwright patch update. Co-authored-by: renovate[bot] <renovate[bot]@users.noreply.github.com>
This PR turns report/export JSON into explicit v1 contracts for programmatic consumers. It adds shared schema, pricing provenance, and project identity metadata to usage daily and activity report outputs, and introduces a daemonless `agentsview export sessions` summary export for headless analytics. The session export is content-free, supports JSON/NDJSON, and includes per-session usage, model, cost, project, worktree, branch, machine, timestamp, and classification metadata without transcript content. Pricing provenance is centralized under `internal/export`: reports use a resolver-derived block with source/table metadata, RFC 8785-style digest, fallback indicators, `cost_source`, and a bounded per-model effective rates map. Source-reported costs are marked so consumers know when token-times-rate recomputation is not expected, and reasoning tokens are handled as output-rate billing breakdowns. Project identity now persists raw observations at sync/import time and recomputes stable identities at export time. Remote-backed identities use normalized network remotes with `sha256:` keys; path-backed fallbacks remain explicit and machine-local. The identity store is preserved through resync and mirrored through PostgreSQL/DuckDB so CLI and HTTP exports stay aligned across backends. The new session-summary export adds stable watermark/keyset pagination, cursor-reset signaling, `--all`, NDJSON meta rows, root/child and automation filtering, and shared pricing/project metadata. Existing usage/activity payloads stay additive: metadata lands as sibling blocks, and daily breakdown arrays are pinned as arrays rather than omitted. Docs now describe the v1 contract rules, pricing digest input, project identity derivation, cursor behavior, session-export limits, and default exclusion caveats. Golden fixtures pin usage daily, usage daily with breakdowns, activity report, and session export JSON/NDJSON shapes. Stale `docs/superpowers` design notes were removed, and the shared contract package was renamed from `internal/exportcontracts` to `internal/export`. Reviewers should focus on: - shared DTO/resolver code in `internal/export` - project identity capture, fallback, resync preservation, and mirror-backend persistence - session summary export query/cursor behavior in `internal/db/session_export.go` and `cmd/agentsview/export.go` - pricing provenance coupling across SQLite, PostgreSQL, and DuckDB usage/activity paths The main tradeoff is landing the related export-contract issues together so field names and semantics stay shared across surfaces. This intentionally does not add redaction flags or per-row pricing provenance: raw project paths/remotes are emitted by default, and pricing provenance remains report-level with a bounded per-model map. Co-authored-by: Wes McKinney <wesm@users.noreply.github.com>
The insights generator can analyze agent activity, but the request path is date-window scoped, so starting from a single session still asks the backend for a sessions-list prompt. That makes the top-bar workflow in kenn-io#445 impossible without manually narrowing dates and still getting surrounding sessions in the prompt. This adds a session-scoped path to the existing insights generation flow instead of creating a parallel endpoint. The session breadcrumb starts an `agent_analysis` task with the current session id, the server validates that session scope only applies to agent analysis, and the prompt builder switches to a single-session branch that reads the session, messages, timing, and usage through the existing store interface. The storage shape stays unchanged for this pass. The PR keeps the generated insight pipeline, SSE stream, task handling, and ordinary date-range analysis intact, and limits the new behavior to the missing session scope. Closes kenn-io#445 Co-authored-by: Rod Boev <rodboev@users.noreply.github.com>
Closes kenn-io#1001. Adds a Korean (`ko`) locale alongside the existing English and Chinese catalogs. ## What changed - **`frontend/messages/ko.json`** (new): full Korean catalog, key set aligned 1:1 with `en`. Plural messages mirror the Chinese catalogs' single-variant (`countPlural=other`) structure because Korean has no plural distinction; all `{placeholder}` tokens and escaped braces are preserved verbatim. - **`project.inlang/settings.json`**: `ko` added to `locales`. - **`src/lib/i18n/index.ts`**: `ko` added to `SUPPORTED_LOCALES`; `normalizeLocale` now matches `ko` and `ko-*` browser tags. - **`src/lib/components/settings/LanguageSettings.svelte`**: Korean option added to the language picker. - **`messages/{en,zh-CN,zh-TW}.json`**: new `settings_language_korean` label key, keeping all catalogs in sync per the `AGENTS.md` localization contract. - **`src/lib/i18n/i18n.test.ts`**: extended for the locale list, `ko` normalization, key alignment across all catalogs, and `ko` rendering / plural-variant selection. ## Where to look The translation quality and the plural-structure parity with the zh catalogs are the main review surface. `normalizeLocale` follows the same shape as the existing `zh-CN` / `zh-TW` branches. Co-authored-by: Leuconoe <Leuconoe@users.noreply.github.com>
…io#1006) ## Summary The Activity dashboard's Total Cost diverged from `agentsview usage daily` (the authoritative daily figure) whenever subagent sessions carried usage. The report's candidate-session query used the default analytics relationship exclusion, which drops `relationship_type = 'subagent'` rows, while `GetDailyUsage` never filters by relationship type — it relies on per-row usage dedup instead. On a real archive, one day showed ~$177 on /activity vs ~$206 from usage daily, with whole models missing from the activity breakdown. All three backends (SQLite, PostgreSQL, DuckDB) now set `IncludeSubagents = true` in `GetActivityReport`, the same opt-in that `GetAnalyticsSummary` already uses for token/cost aggregates. Fork rows stay excluded because they replay a root session's messages and would double-count; the new tests pin both behaviors, and the cross-backend parity fixture now includes a subagent and a fork session. Verified end-to-end against a copy of a real 22 GB archive: the activity report and `GetDailyUsage` now agree exactly (cost and output tokens) for the same day and timezone. One observable side effect: the report's `sessions` count and `by_session` table now include subagent rows. This restores the intended cost semantics rather than changing row meaning, so `ActivityReportSchemaVersion` stays at 1; the docs note the subagent-counting behavior. Where to look: the one-line filter change in each backend's `GetActivityReport` (`internal/db/activityreport.go`, `internal/postgres/activityreport.go`, `internal/duckdb/activityreport.go`); everything else is tests and docs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Wes McKinney <wesm@users.noreply.github.com>
…n-io#1008) Fixes a sustained automountd/opendirectoryd CPU storm on macOS caused by project-identity code probing the local filesystem with paths that belong to other machines. On darwin, `/home` (and `/net`, `/Network/Servers`) are automounter triggers: merely stat'ing a path under them wakes `automountd`, which resolves the map through `opendirectoryd`, and negative results are not cached. An archive holding tens of thousands of sessions synced from a remote Linux machine (thousands of distinct `/home/...` cwds and stored root paths) turned routine work into a continuous syscall storm — measured live as automountd pinned at ~100% and opendirectoryd at 1.3–3.6 cores, dropping to exactly zero with the daemon stopped; `fs_usage` showed agentsview threads issuing `lstat64`/`readlink` on `/home` at ~1,500 calls/sec. Two probe paths are fixed: 1. **Sync-side discovery** (`cachedProjectIdentity` → `discoverLocalGitIdentity`): ran `filepath.EvalSymlinks` plus a git-root walk on every session's cwd, including foreign-machine sessions, on every identity-cache miss (one-minute TTL). Now gated on the observation's machine matching the engine's own machine identity, next to the existing `idPrefix`/`pathRewriter` remote-import gate. Foreign-machine observations keep their raw cwd with no git enrichment, which is what they should have carried all along. 2. **Read-side resolution** (`export.NormalizeRootPath`, `export.NormalizeStoredRootPath`, and the legacy discovery in `db.BuildProjectIdentityMap`): every activity/usage/export request rebuilds the project-identity map, and each stored `/home/...` root was re-resolved through `EvalSymlinks` per request. A new `export.IsAutomountNamespacePath` predicate skips filesystem resolution for macOS automounter namespaces in all three places (plus the sync-side discovery, for local sessions with such cwds). On darwin these resolutions always failed and fell back to the cleaned path, so skipping them produces identical identity keys minus the syscalls; other platforms are unaffected. With both fixes deployed on the same archive under identical daemon + embedding load, automountd and opendirectoryd burn dropped to zero (0.00s and 0.13s CPU over a 2-minute sample, from ~1.75 and ~3.3 cores). Rows already written with locally resolved roots for foreign sessions are left as-is; they age out through normal observation upserts. Supersedes kenn-io#1007. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Wes McKinney <wesm@users.noreply.github.com>
…#1015) ## What changed - Drops unchanged OpenCode-family shared-container sessions after parse using session mtime, `file_hash`, and data version. - Stores the OpenCode storage fingerprint on SQLite-backed OpenCode sessions too, so same-mtime content changes are still re-emitted. - Adds regressions for unchanged containers, one changed row among unchanged siblings, and same-mtime content changes. - Hardens a few test fixtures that depended on host temp paths, PATH coreutils, or global git signing. ## Why OpenCode-format providers fan one SQLite or storage container into virtual per-session rows. Without the post-parse unchanged drop, stable sessions were rewritten on every full sync, bumping `local_modified_at` and amplifying remote push writes. ## Notes for reviewers - The sync behavior change is in `dropUnchangedSharedSQLiteResults`. - The fixture-hardening commit is separate from the sync fix. Co-authored-by: Phillip Cloud <cpcloud@users.noreply.github.com>
) ZCode sessions were invisible because agentsview did not have an agent type, provider, parser, or sync route for the SQLite database reported in kenn-io#1003. This adds `AgentZCode` on the existing DB-backed provider path. The provider resolves either `.zcode/cli/db` or `.zcode/cli` to `db.sqlite`, reads the reported `session` and `model_usage` tables, and emits metadata-only sessions plus usage events. Freshness uses the same effective mtime for provider fingerprints and stored session file info, combining session timestamps, usage timestamps, and the SQLite DB/WAL/SHM mtimes so usage-only updates stay visible. This covers metadata and usage from the issue's reported schema. Live ZCode compatibility and transcript-message storage need a sample DB, local install, or upstream schema source. Closes kenn-io#1003 Co-authored-by: Rod Boev <rodboev@users.noreply.github.com>
…it citations (kenn-io#999) Adds opt-in semantic/vector search over conversation content alongside the existing substring/regex/FTS modes, and gives every content-search match — in every mode, on all three backends — a conversation-unit citation. ## Semantic search - `session search --semantic` and `--hybrid` (reciprocal-rank fusion of the vector and FTS legs), plus `--scope top|all|subordinate` to control whether subordinate evidence (sidechain runs, subagent/fork sessions) is shown; subordinate hits are rank-penalized and annotated, never silently hidden. - Embedding documents are run-grouped: each user message is one document, and each unbroken run of assistant/tool messages between user turns is concatenated into one document (~25x fewer assistant-side documents than per-message embedding). Semantic hits anchor on the message containing the best-matching chunk's center and carry the run's ordinal span. - `embeddings build/list/activate/retire` manage the index: through the daemon when one is running, directly (flock-guarded) otherwise. `serve` wires a debounced after-sync scheduler when `[vector]` is enabled. - Embeddings come from any OpenAI-compatible endpoint (`[vector]` in config.toml; Ollama quickstart in the docs). An optional `input_suffix` appends a client-side terminator to every embedded text for models that need one (e.g. Qwen3-Embedding's `<|endoftext|>` under llama.cpp); it joins the generation fingerprint. - `session messages --around N --before/--after --role` retrieves context windows around any hit on all three backends; `session search --context N` inlines them. ## Conversation-unit citations Every match now carries `ordinal_range: [start, end]` — the conversation unit enclosing the anchor — plus `subordinate`, `relationship`, `parent_session_id`, and `is_sidechain`. `ordinal` remains the exact matched message in every mode. - Row cardinality is mode-specific by design: lexical modes stay grep-like (one row per matching source row), semantic returns one row per embedded unit, hybrid one row per unit with exact-match anchors. - Lexical and hybrid unit-less rows derive their unit structurally from the messages/sessions tables — deterministic, identical on SQLite/PostgreSQL/DuckDB, and independent of whether a vector index exists. A property test pins derivation to exact equivalence with the embedding reducer's unit spans. - Derivation is post-scan and O(page): batched correlated point lookups with run sharing, ~0.4-0.65ms added per 50-hit page on the gated benchmarks. - Surfaces: CLI renders `#start-end @anchor` with a `sub` marker; MCP `search_content` carries the same fields; the OpenAPI schema and generated client are updated. ## SQLite DSN hardening Read-only connections were silently read-write: mattn/go-sqlite3 ignores `mode=ro` without a `file:` URI scheme. Fixed for sessions.db, vectors.db, and every foreign-app parser DB (which were also being converted to WAL on read). The fix exposed and fixed a WAL close-ordering bug in the resync swap flow. Paths are percent-escaped; read-only enforcement is pinned by tests. ## Architecture Vectors live in a separate `vectors.db` (SQLite + sqlite-vec via go.kenn.io/kit), a mirror keyed by resync-stable doc keys with generations fingerprinted by model/dimension/unit-scheme config; a mirror schema version gates cross-version reads (rebuild-required surfaces as 501 with remediation) and resets stale mirrors on writable opens. Staleness, first-build progress, and endpoint outages surface as distinct errors across CLI/HTTP/MCP. See `docs/semantic-search.md` (usage) and `docs/semantic-search-internals.md` (unit model, doc keys, derivation invariants, fusion, error taxonomy). ## Where to look - `internal/db/messages.go`, `internal/db/unit_range.go` — run reducer and shared unit-range derivation (the correctness core; reducer-equivalence property test). - `internal/vector/` — mirror, generations, chunk-anchor resolution, build orchestration, encoder, search. - `internal/db/search_content*.go` — semantic/hybrid modes, unit-granularity fusion, scope filtering, lexical citation enrichment. - `internal/postgres/unit_range.go`, `internal/duckdb/unit_range.go` — SQL-only backend seams over the shared resolvers. - `cmd/agentsview/embeddings.go`, `embed_scheduler.go` — CLI group, serve wiring, scheduler. - `internal/vector/encoder.go`, `internal/vector/build.go` — build throughput: requests use `encoding_format: "base64"` (~4x smaller responses, with transparent float fallback for servers that reject or ignore the field), and a `[vector.embeddings] concurrency` key (default 4) embeds documents in parallel via kit's new `FillOptions.Concurrency` (kenn-io/kit#27; go.mod pins that PR's commit and should move to a tagged kit release before merge). Saves stay serialized, preserving the single-writer model. Sequential float-JSON requests left builds round-trip-bound against remote endpoints; measured on a slow WireGuard link, these two changes took a full-archive build from ~46 to ~700 chunks/min. - Frontend diff is regenerated API-client output only; the web UI remains FTS-only in this release. ## Limitations - Semantic/hybrid search is SQLite-archive only; `pg serve`/DuckDB validate and report it unavailable (citations work on all three). - Metadata filters post-filter the vector leg (over-fetch mitigates recall loss); narrow scopes can under-fill a page past the batched FTS-leg cap. - Citation derivation adds ~19-21% to content-search page latency (sub-millisecond absolute); levers (covering index, statement cache) documented but not pulled. - Draft: kept open for real-world exercise before merge. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Wes McKinney <wesm@users.noreply.github.com>
Fixes date-range picker reconstruction so pinned fixed ranges that exactly match calendar periods come back as calendar selections instead of falling through to custom. This preserves Calendar Day/Week/Month state for analytics, usage, trends, and insights after the stores persist only concrete from/to bounds.
The implementation adds one shared helper in `rangeSelection.ts` that validates local ISO date strings and recognizes exact day, ISO Monday-Sunday week, and full calendar month bounds before the custom fallback. Relative presets and all-time handling still take precedence where they already did. The remaining ambiguity is intentional: if a user-defined custom range exactly equals a canonical calendar period, the picker now shows the canonical Calendar selection because the stored `{from,to}` alone cannot preserve the original mode.
Tests cover pinned calendar-week reconstruction, exact-month reconstruction, and keep the Trends custom fixture on a noncanonical span so it continues to assert the actual custom path. Reviewers should start in `frontend/src/lib/components/shared/rangeSelection.ts`.
Co-authored-by: Phillip Cloud <cpcloud@users.noreply.github.com>
Fixes the pre-existing silent no-op where clicking an agent in Usage attribution updated local exclusion state, but the next usage requests did not send `excludeAgent`. The store now includes agent exclusions in the shared request params and counts them in `hasActiveFilters`, so the API sees the same filter state the UI shows. The Usage toolbar also gets the Agent dropdown next to Project and Model. It uses the existing dropdown pattern, which makes agent exclusions visible and reversible without relying on attribution-row clicks. This stays scoped to Usage agent filtering and does not depend on the branch-dimension work.  Reviewers should start with `frontend/src/lib/stores/usage.svelte.ts`; `UsagePage.svelte` is the toolbar wiring, and the added component/store tests cover the request params plus the attribution-click path. Fixes kenn-io#976. Co-authored-by: Prateek Rungta <prateek@users.noreply.github.com>
AgentsView now reads Qoder and QoderWork transcripts from `~/.qoder/projects/` and `~/.qoderwork/projects/`. The files in kenn-io#1011 use Claude-style JSONL, so the provider runs them through the existing Claude parser and retags the parsed sessions as Qoder instead of carrying a separate parser. The provider discovers main transcripts and `subagents/agent-*.jsonl` files, applies `*-session.json` title, cwd, and fork metadata, force-replaces Qoder parses so Claude-style streamed chunks update stored rows, and keeps subagent links stable across forked rows. Sync path classification and the frontend agent label are wired up so configured Qoder directories behave like the other file-backed agents. QoderCLI stays out of scope because kenn-io#1011 describes only `ai-stats/` there, with no transcript files to import. kenn-io#1000 is covered by this Qoder and QoderWork support; kenn-io#1011 has the concrete layout and samples used here. Reviewer entry points are `internal/parser/qoder*.go` for discovery and retagging, `internal/sync/qoder_test.go` for sync behavior, and `frontend/src/lib/utils/agents.ts` for the display label. Closes kenn-io#1011 Co-authored-by: Rod Boev <rodboev@users.noreply.github.com>
The applyConfigTOML function TOML parsing struct was missing the Port field, causing "port = 19090" in config.toml to be silently ignored. The field was parsed by the TOML decoder but never applied to the Config. Add the Port field to the parsing struct and apply it when non-zero. Co-authored-by: Mr Koala <Mr-Koala@users.noreply.github.com>
…io#993) Worktree mappings currently preserve project names only when each repository has its own explicit path-prefix row. That works for a few deleted worktrees, but it does not scale for layouts such as `{repo}.worktrees/{branch}` where the canonical project is already present in the path segment. This adds a layout mode to the existing worktree mapping row instead of creating a separate template system. Existing rows stay explicit and keep their current behavior. A new `{repo}.worktrees/{branch}` layout lets one parent-directory mapping derive the project from `service.worktrees` style directories, while the existing longest-prefix precedence still lets a more specific explicit mapping override the generic layout. The resolver stays in the worktree mapping layer, and the settings API and UI only adapt that row shape. Scope is limited to the local worktree mapping path; parser behavior, backend sync semantics, and unrelated project inference remain unchanged. Fixes kenn-io#582 Co-authored-by: Rod Boev <rodboev@users.noreply.github.com>
…o#1014) ## Why Push candidacy was still sensitive to local stat churn. A no-op local row rewrite could bump `local_modified_at`, and a file touch could bump `file_mtime`, which made unchanged sessions look dirty and rewrote mirror rows repeatedly. The user-visible identity for a pushed session is its content, counters, metadata, source path, and content-derived `file_hash`. Filesystem stat fields are useful mirror data, but they should not decide whether a session changed. ## What Changed - DuckDB fingerprints now keep `file_path` and `file_hash`, but ignore `file_size`, `file_mtime`, `file_inode`, `file_device`, and `local_modified_at`. - PostgreSQL fingerprints now keep `file_path` and `file_hash`, but ignore `file_mtime` and `local_modified_at` for parity with DuckDB. - PostgreSQL also folds local dependent-row fingerprints into session candidacy, so same-count message edits, pin changes, and secret-finding changes still push even when `file_hash` is absent or stale. - PostgreSQL preserves prior boundary fingerprints when successful runs skip unchanged sessions, so stat-only churn stays skipped after empty and mixed push cycles. - Coverage checks stat-only churn, watermark advancement, content changes, the PostgreSQL same-count message-edit case, alias path changes, and preserved skipped-session fingerprints. ## Behavior Notes - `localSessionSyncMarker` is unchanged, so volatile timestamps can still bring sessions into the incremental candidate list. - Mirror write paths are unchanged. When a session really pushes, the stat columns are still written. - The accepted tradeoff is that mirror stat columns can stay stale while session content is unchanged. ## Upgrade Notes Existing stored fingerprints were computed with the old field set, so the first push after upgrade will re-push sessions once. After that, unchanged stat churn should no longer keep sessions hot. ## Review Pointers - DuckDB candidacy: `internal/duckdb/sync.go` - PostgreSQL candidacy and finalization: `internal/postgres/push.go` and `internal/postgres/push_fingerprint.go` - Regression coverage: `internal/duckdb/sync_fastpath_test.go` and `internal/postgres/push_test.go` Co-authored-by: Phillip Cloud <cpcloud@users.noreply.github.com>
The PR adds support for the RooCode VSCode extension. Even though the RooCode project has been shut down, this addition to agentsview is useful for historic session analysis and search, and could be a baseline for adding support for Kilo IDE VSCode extension (which was a RooCode fork until the rewrite) and the new ZooCode fork. Co-authored-by: Stephen Cross <scross01@users.noreply.github.com>
…-io#1177) AgentsView now treats Hermes named profiles as live archives rather than a startup-only list of transcript directories. Long-running servers discover profiles created after initialization and import sessions plus `state.db` metadata, including database-only sessions and WAL-only updates. HTTP remote sync now publishes each Hermes `state.db` as a standalone SQLite online-backup snapshot whose manifest identity includes uncheckpointed changes. Full and delta transfers therefore cannot mix a main database with a different WAL generation, while target resolution remains confined to transcripts and database files instead of credential-bearing profile contents. Hermes overrides retain replacement semantics, trailing-slash handling, profiles-container parity, and supported flat transcript roots across local and remote resolution. SSH remote sync remains available for compatibility, but it is deprecated and receives only critical fixes. Remotes with Python 3 and SQLite backup support get the same coherent database snapshots; every database snapshot omitted because that support is unavailable or snapshotting fails emits a path-specific warning, while transcripts and unrelated agent data continue transferring. The CLI warns once per process, and the help and remote-sync documentation direct new configurations to HTTP. Co-authored-by: jahabdank <jahabdank@users.noreply.github.com>
Renovate currently treats the frontend TypeScript pin as eligible for a 7.x major update, so closing the bot PR would only cause it to be recreated. TypeScript 7 is not yet an intended migration for agentsview. This constrains Renovate TypeScript candidates to versions below 7. TypeScript remains exactly pinned in package.json, and Renovate can continue proposing 6.x maintenance releases without bundling a premature compiler major into the JavaScript dependency group. Co-authored-by: Wes McKinney <wesm@users.noreply.github.com>
…nn-io#1196) Fixes kenn-io#1195. The StatusBar sync-label test computed its expected title with `toLocaleString(undefined, ...)`, which resolves to the OS locale, while the component formats the title through the i18n `formatDateTime` using the Paraglide app locale (`en` in the test environment). The two sources only agree on English-locale systems, so the full suite failed on any non-English machine (observed on Korean-locale Windows; reproduced on current `main`). The expected value is now computed through the same `formatTimestamp` path the component uses, which keeps the assertion focused on the wiring contract and makes it independent of the host locale. The formatting behavior itself is already covered by the i18n unit tests, including the locale-selection cases in `i18n.test.ts`. This was the only `toLocaleString`-based expectation in the test suite. Co-authored-by: bangddong <bangddong@users.noreply.github.com>
Grok Build's public source shows that resumable local sessions span current type-based rows, legacy role-based rows, and mixed histories, with additional compatibility rules for reasoning, backend tool calls, synthetic prompts, and session metadata. The previous parser handled only a subset of that persisted contract. This aligns parsing with xai-org/grok-build at commit 7cfcb20d2b50b0d18801a6c0af2e401c0e060894. It preserves supported current and legacy transcript shapes, mirrors reasoning association and backend-call deduplication, and applies the source-defined title, fork, workspace, and context-token semantics. Sanitized golden sessions are generated through Grok Build's own Rust serializers and downgrade binary, while the Go expectations remain independently authored so the producer is not also the test oracle. <sup>generated by a clanker</sup> Co-authored-by: Marius van Niekerk <mariusvniekerk@users.noreply.github.com>
…enn-io#1194) Fixes kenn-io#1086. Date bounds materialized from a rolling window (`window_days`) were saved into the `session-filters` localStorage entry and restored verbatim on the next launch. Once yoked dates could no longer overwrite them — e.g. after the v1 storage migration disabled them — the range stayed pinned to the day it was saved and newer sessions silently disappeared from the sidebar and analytics. The sessions store now tracks the provenance of its date bounds as `dateFiltersWindowDays`: bounds applied from a rolling panel state (`App.svelte`, `AnalyticsPage.svelte`) or a `window_days` deep link (`initFromParams`, validated with the route layer's shared `parseWindowDaysParam`) carry their rolling intent, while explicitly chosen fixed ranges carry `null`. Persistence stores the intent rather than the materialized dates: the saved entry keeps `windowDays` and blank bounds, and `loadSavedFilters` rematerializes the bounds against the current date, so a rolling filter survives restarts and keeps rolling forward instead of pinning or vanishing. `applyPanelDateFilters` persists immediately, so a provenance flip between fixed and rolling with identical materialized bounds — which does not register as a filter change and may never trigger a `load()` — still updates the saved entry. Wholesale filter resets clear the intent, and the store is now the sole writer of session date bounds (`clearSessionDateFilters` removed as dead code). Already-poisoned installs are also repaired: saved entries are stamped with a storage version, and unversioned (pre-provenance) entries have their date bounds dropped once on load and are rewritten in the new format, mirroring the yoked-dates v1 migration. The check is deliberately `!==` rather than `<` so a downgraded newer format is also not trusted; dropping bounds is the fail-safe direction in both. Other persisted filter fields are preserved. Reviewers should look at `saveFilters`/`loadSavedFilters`/`applyPanelDateFilters` in `sessions.svelte.ts` and the two call sites. Known cosmetic note: a persisted one-day rolling window rematerializes as `date_from`/`date_to` rather than the single `date` param; both filter identically. Co-authored-by: bangddong <bangddong@users.noreply.github.com>
Renders the per-match message timestamp in the human output of `agentsview session search`, so results can be associated with a session by when it happened rather than only by the snippet text. The timestamp was already populated on every match in all search modes and already emitted in `--output json` — the CLI change is presentation-only. - Table format gains an AGE column immediately after MATCH (before the optional SCORE): relative under a week (`3h`, `5d`), `Jan 02` within the current year, `Jan 2025` for prior years, em dash when the timestamp is missing or unparseable. The year appears for prior years because search spans the whole multi-year archive, unlike the resume-oriented `session list`. - The `--context` record format gains the same age token on the match line, between the ordinal/score markers and the project. - The relative buckets are extracted from `humanizeSessionAge` into a shared `humanizeAgeRelative` core; `session list` output stays byte-identical, and JSON output is unchanged. - The renderers now take an injected clock (captured once per render) so tests pin output against a fixed time. Following review, the branch also normalizes search-content timestamps at the backend boundary: the DuckDB quack store and the PostgreSQL read store (substring/regex scans and the shared semantic/hybrid hit enrichment) were emitting SQL-style `timestamp::text` / `CAST(... AS TEXT)` strings, which the AGE parser — and any other RFC3339 consumer — rejects. Both backends now scan the raw timestamp column and format RFC3339Nano UTC, matching the SQLite backend and the stores' existing session-timestamp helpers (`formatDBTime`, `FormatISO8601`), with regression tests asserting the returned instant equals the inserted one. Where to look: `cmd/agentsview/session_search.go` (new `humanizeMatchAge` helper and the two renderer changes), `cmd/agentsview/session_list_render.go` (the extraction), `internal/duckdb/store.go` and `internal/postgres/search_content*.go` (the timestamp normalization). Co-authored-by: Matthew Jacobs <mjacobs@users.noreply.github.com>
Grok's provider reads session metadata, signals, and chat history while leaving the cumulative usage ledger in `updates.jsonl` outside freshness tracking and parsing, so daily and session usage remain empty. This adds `updates.jsonl` to Grok companion freshness and converts the latest valid ledger snapshot into the existing usage-event model. Per-model rows remain the accounting authority when present, cache reads are separated from the producer's full input count, later smaller snapshots replace earlier values instead of being summed, and `signals.json` keeps ownership of peak-context reporting. Summary, signal, and transcript behavior stay unchanged when no usable snapshot exists; existing model pricing still applies when Grok does not report a cost. The parser keeps the last valid `params.update.usage` object, emits one event per `modelUsage` entry or one summary-model event when the map is empty, maps each selected object's `costUsdTicks` onto `CostUSD`, and leaves `CostUSD` nil when the field is absent so catalog pricing remains the fallback. The sync integration now records input `510`, cache-read `131456`, output `326`, reasoning `122`, cost `0.0424128`, peak context `4096`, and model `grok-4.5-build` for the issue-shaped payload. Watch-plan coverage pins `updates.jsonl` alongside the existing summary, signal, and transcript globs. Parser tests also cover a malformed trailing line retaining the latest snapshot and an updates file with no usable usage object producing no events. Closes kenn-io#1193 Co-authored-by: Rod Boev <rodboev@users.noreply.github.com>
kenn-io#1169) Use Copilot CLI `session.shutdown.totalNanoAiu` as authoritative USD cost through the existing generic `cost_usd` and `cost_source` model only for sessions starting on or after June 1, 2026, when GitHub moved new sessions from premium-request billing to usage-based AI Credits. Older sessions, sessions without reported billing, and other Copilot-family agents retain catalog-priced estimates. The final cumulative shutdown value wins, including zero. Per-model token analytics remain intact, while model-filtered cost stays estimated because a session-level charge cannot be allocated reliably among models. SQLite, PostgreSQL, and DuckDB preserve matching aggregation behavior, including cost-only shutdown carrier rows that do not surface as token data, models, or breakdown rows. Pricing provenance records `mixed` or `reported` when an authoritative session total is selected instead of incorrectly reporting `computed`. Existing usage contracts remain unchanged: the Copilot AI Credits summary card, usage-daily `copilotAICredits` totals field, session `ai_credits` field, CLI `AI Credits` line, localization keys, and generated API types are retained. Credits derive from the final selected USD cost, whether reported or catalog-computed. No database schema change, Copilot-specific cost column, migration, compatibility probe, dual-read path, or usage-daily schema-version change is introduced. The contract remains at `schema_version: 2`. Parser data version 69 re-parses existing sessions so eligible sessions created since June 1, 2026 receive reported `cost_usd` and `cost_source`; older sessions remain catalog-priced. Orphan archive-copy behavior remains unchanged from `main`. Co-authored-by: Erik Krogen <xkrogen@users.noreply.github.com>
Provider parsers rely on upstream on-disk formats whose token, cache, reasoning, and monetary-cost fields can change independently of Agentsview. Those assumptions were previously scattered across implementation details and fixtures, making format investigations repetitive and leaving authority boundaries unclear for closed-source products. This adds a maintained provider-by-provider provenance inventory with immutable source revisions where available, first-party documentation where source is closed, and explicit exhausted-search results otherwise. It also records known accounting limitations, enforces registry coverage and evidence shape, and directs future provider implementers to reverify the inventory when formats or usage behavior change. Grok remains temporarily excluded while its separately owned format-alignment work lands. <sup>generated by a clanker</sup> Co-authored-by: Marius van Niekerk <mariusvniekerk@users.noreply.github.com>
Builds on kenn-io#1159's extraction store and client with the manager layer that drives model-backed recall extraction end to end. ## Manager (`internal/recall/extract`) - `ExtractCandidates` selects eligible sessions in SQL: ended past the quiet period, not automated, not trashed, zero secret findings, scanned under the current rules versions, and non-empty. The privacy predicates are not configurable, and the same checks guard explicit single-session runs so no path can feed an excluded session to the model. Failed sessions retry behind an updated_at-indexed backoff arm. - `Manager.RunPass` distills each candidate unit by unit, checkpointing a resumable cursor after every unit. Unit output commits under an in-transaction guard that re-verifies the session snapshot, eligibility, and absence of secret findings atomically with the insert; every Go-side check is advisory over that guard. - Failures mark the session for retry after a per-session backoff instead of aborting the pass — except endpoint-scoped failures (401/403/404/405/415/501, refused redirects, schema-violating responses), which abort the pass with rows left pending so a broken endpoint cannot burn one doomed model call and a backoff per session. Failure transitions never advance the coverage stamp outside the transaction that records the failure. - Units the model rejects as too large (context-overflow 400 or HTTP 413) or cannot answer completely (persistent truncation) are halved recursively down to the split floor. - Model responses are bounded locally (entry count, field lengths, mirrored as maxItems/maxLength in the request schema) and persisted error text is capped, so a hostile endpoint cannot balloon the archive. - Entry ids are deterministic (`sha256` of generation fingerprint, session, unit, entry position), and the bulk insert skips existing ids, so replays after crashes or digest resets dedupe instead of duplicating. - Entries carry evidence rows with the unit's message-ordinal range, session context (project, cwd, branch, agent), the generation fingerprint as `source_run_id`, and `unreviewed_auto` review state. While a generation is building, entries stage as `archived` and never serve. - Privacy retraction runs on every scheduled pass, before any model work (so extraction failures cannot defer it) and again after the loop: sessions since trashed, flagged automated, or carrying findings get their generated entries deleted across all generations and their progress rows removed. ## Activation A generation auto-activates once everything eligible is done and it has produced entries; explicit `Activate` refuses an empty generation. The activation transaction re-verifies coverage — no eligible session pending/partial, unextracted, or with coverage stamped before its latest transcript write or under superseded scan rules — and aborts with a typed error rather than retiring the served corpus around a gap. It clears the staged output and progress of any session no longer fully eligible (trashed, reopened, awaiting rescan, or gone) so nothing stale serves and nothing strands archived, then promotes the rest atomically while retiring the previous generation. ## Credential handling Endpoint URLs can carry credentials in userinfo, query values, bare query tokens, fragments, and path segments. `config.RedactedEndpoint` masks all of these fail-closed (only `api-version` and known API-surface path vocabulary stay visible) for every error, log line, and stored failure row. Response bodies are attacker-influenced and can reflect the request: when the endpoint URL carries any credential material, all endpoint-provided diagnostic detail is withheld rather than scrubbed (re-encodings defeat literal replacement); credential-free endpoints keep a control-stripped, length-capped excerpt. Redirects are never followed — a redirect would replay transcript content to an attacker-chosen destination — and a refused redirect aborts the pass. ## Secrets integration Transcript mutations revoke the session's secret-scan stamp in-write (`rulesAlgorithmVersion` 7), so appended content cannot ride an older scan's approval. The manager additionally rescans outbound text against the full ruleset before sending and fails closed on any match despite a current stamp. ## Config (`[recall.extract]`) Model identity (`model`, `deployment`), named servers (transport only — moving a deployment to a new address does not orphan the corpus), `quiet_period`, `backstop_interval`, `failure_backoff`, `max_window_chars`, `max_tokens`, prompt profile/override-dir selection, and request-shape overrides. Validated at load, and the resolved request shape is validated at manager construction, so a bad profile fails setup before any progress rows exist. Disabled section stays inert. ## Daemon scheduler Mirrors the embed scheduler: sync completions debounce into incremental passes; backstop ticks run full passes that revisit done sessions so grown transcripts are topped up via content-digest reset; with the backstop disabled a catchup ticker keeps incremental passes running. Every daemon lifetime starts with a full pass (deferred work survives daemon restarts), the pending startup pass and every running pass hold an idle-work lease so a detached daemon neither reaps itself mid-pass nor before its first pass, and no pass starts once the daemon is draining. Session-mutating server routes (trash, restore, delete, empty-trash, secret scan) notify the scheduler so eligibility changes are picked up without waiting for sync activity. ## CLI `recall extract` becomes a parent command: - `run [--session <id>] [--full] [--limit N]` — one pass; `--session` bypasses the quiet period but never the privacy filters - `status` — coverage per state, unit progress, entry count, generation list - `activate` / `retire <fingerprint> [--force]` — with the store's refusal guards - `doctor` — prints the resolved model/server/profile/fingerprint and makes one probe call whose deadline derives from the configured server timeout - `preview --session <id>` — the previous `--dry-run` chunk preview; the legacy `extract --session --dry-run` flags still work as a silent fallback Manual write commands refuse while a daemon owns the archive, since an enabled daemon runs passes itself. Design notes live in `docs/internal/recall-extraction.md`. Note: `TestDoSyncConfiguredFullUnifiedHTTPUsesManifestDeltaAndOrderedProgress` fails on this machine on clean `origin/main` as well; unrelated to this change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Wes McKinney <wesm@users.noreply.github.com>
The session analysis sidebar now shows the repository label and trace-recorded working directory above the timing metrics, so users can confirm which checkout produced the session they are inspecting. The pane reuses the already-hydrated session detail rather than adding another request or expanding the timing API. Long worktree paths truncate within the 320px sidebar while retaining the complete value in a title tooltip. Hovering a repository or worktree row reveals a copy control, keyboard focus reveals the same control, and touch devices keep it visible. Repository context remains available if timing data fails to load, and all labels and copied-state feedback are localized across every supported catalog. The branch also includes the concise design and implementation plan used for the first draft. <sup>generated by a clanker</sup> Co-authored-by: Marius van Niekerk <mariusvniekerk@users.noreply.github.com>
…nn-io#1204) Opening a command-palette message-search result can leave the session breadcrumb without its project or display name. The palette selected the session through the sessions store before committing the route, and a target absent from the current sidebar index page loses its hydrated row when the index reload rebuilds the list, with nothing re-establishing it afterward. This makes session opening route-first and hydration self-healing. The palette's search and recent results and the insights evidence links now commit the URL and let App's deep-link effect own selection and hydration, exactly as direct deep links do; that effect now tracks the routed session's hydration state, so a sidebar reload that drops or de-hydrates the routed row triggers a re-fetch. The sidebar rebuild keeps the active session's hydrated row when the incoming index page omits it, and pagination moves that row into its real position instead of duplicating it; `navigateToSession` joins an in-flight fetch for the same session instead of restarting it. Direct deep links, sidebar clicks, keyboard navigation, and `?msg` handling keep their current paths, and route exit still cancels all session-route reads unconditionally. A Playwright regression drives the real palette flow against an index page that omits the target; it fails on the previous ordering with an empty breadcrumb and passes with this change. The interaction steps came from @tekumara's issue report.  Closes kenn-io#1190 Co-authored-by: Rod Boev <rodboev@users.noreply.github.com>
…#1214) This PR contains the following updates: | Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) | |---|---|---|---| | [dompurify](https://redirect.github.com/cure53/DOMPurify) | [`3.4.11` → `3.4.12`](https://renovatebot.com/diffs/npm/dompurify/3.4.11/3.4.12) |  |  | --- ### DOMPurify: `CUSTOM_ELEMENT_HANDLING` bypasses `afterSanitizeElements` for allowed custom elements. [GHSA-c2j3-45gr-mqc4](https://redirect.github.com/advisories/GHSA-c2j3-45gr-mqc4) <details> <summary>More information</summary> #### Details ##### Summary There is a possible hook-policy inconsistency in DOMPurify 3.4.11 involving `CUSTOM_ELEMENT_HANDLING`. When a custom element is allowed via `CUSTOM_ELEMENT_HANDLING.tagNameCheck`, it appears that the element does not go through `afterSanitizeElements` in the same way as a normal element. As a result, an application that relies on `afterSanitizeElements` as a security policy layer to strip sensitive attributes from all elements may see those attributes removed from normal elements but preserved on allowed custom elements. This does not appear to be a direct DOMPurify XSS or a case where DOMPurify directly allows executable payloads. The preserved value is still inert at sanitize time. The issue becomes relevant when the allowed custom element later re-injects that attribute value into an HTML sink such as `innerHTML`, creating a second-order XSS gadget. ##### Details The issue appears to originate from the control flow in `src/purify.ts`: line 1672~1691 ```tsx const _sanitizeDisallowedNode = function ( currentNode: any, tagName: string ): boolean { /* Check if we have a custom element to handle */ if (!FORBID_TAGS[tagName] && _isBasicCustomElement(tagName)) { if ( CUSTOM_ELEMENT_HANDLING.tagNameCheck instanceof RegExp && regExpTest(CUSTOM_ELEMENT_HANDLING.tagNameCheck, tagName) ) { return false; } if ( CUSTOM_ELEMENT_HANDLING.tagNameCheck instanceof Function && CUSTOM_ELEMENT_HANDLING.tagNameCheck(tagName) ) { return false; } } ``` `CUSTOM_ELEMENT_HANDLING` is parsed from user configuration at `src/purify.ts`: line 741~748 ```tsx const customElementHandling = objectHasOwnProperty(cfg, 'CUSTOM_ELEMENT_HANDLING') && cfg.CUSTOM_ELEMENT_HANDLING && typeof cfg.CUSTOM_ELEMENT_HANDLING === 'object' ? clone(cfg.CUSTOM_ELEMENT_HANDLING) : create(null); CUSTOM_ELEMENT_HANDLING = create(null); ``` In particular, `tagNameCheck`, `attributeNameCheck`, and `allowCustomizedBuiltInElements` are copied into the internal `CUSTOM_ELEMENT_HANDLING` object there. During element sanitization, `_sanitizeElements()` checks whether a node is forbidden or not allowlisted at `src/purify.ts`: line 1805~1814 ```tsx /* Remove element if anything forbids its presence */ if ( FORBID_TAGS[tagName] || (!( EXTRA_ELEMENT_HANDLING.tagCheck instanceof Function && EXTRA_ELEMENT_HANDLING.tagCheck(tagName) ) && !ALLOWED_TAGS[tagName]) ) { return _sanitizeDisallowedNode(currentNode, tagName); } ``` If so, it immediately delegates to `_sanitizeDisallowedNode(currentNode, tagName)` and returns its boolean result. Inside `_sanitizeDisallowedNode()`, the custom-element-specific allow path is implemented at `src/purify.ts`: line 1672~1692 ```tsx const _sanitizeDisallowedNode = function ( currentNode: any, tagName: string ): boolean { /* Check if we have a custom element to handle */ if (!FORBID_TAGS[tagName] && _isBasicCustomElement(tagName)) { if ( CUSTOM_ELEMENT_HANDLING.tagNameCheck instanceof RegExp && regExpTest(CUSTOM_ELEMENT_HANDLING.tagNameCheck, tagName) ) { return false; } if ( CUSTOM_ELEMENT_HANDLING.tagNameCheck instanceof Function && CUSTOM_ELEMENT_HANDLING.tagNameCheck(tagName) ) { return false; } } ``` If the node is treated as a basic custom element and `CUSTOM_ELEMENT_HANDLING.tagNameCheck` matches, the function returns `false` immediately at line 1682 or 1689, meaning “do not remove this node”. That early `return false` is significant because control returns directly to `_sanitizeElements()` via the `return _sanitizeDisallowedNode(...)` at line 1813. As a result, the later logic in `_sanitizeElements()` is skipped for that custom element instance, including: - the namespace validation at `src/purify.ts`: line 1816~1826 ```tsx * Check whether element has a valid namespace. Realm-safe check (GHSA-hpcv-96wg-7vj8): use the cached Node.prototype nodeType getter rather than `instanceof Element`, which is realm- bound and short-circuits to false for any node minted in a different realm — letting a foreign-realm element with a forbidden namespace slip past the namespace check entirely. */ const nt = getNodeType ? getNodeType(currentNode) : currentNode.nodeType; if (nt === NODE_TYPE.element && !_checkValidNamespace(currentNode)) { _forceRemove(currentNode); return true; } ``` - the fallback-tag mXSS check at `src/purify.ts`: line 1828~1837 ```tsx /* Make sure that older browsers don't get fallback-tag mXSS */ if ( (tagName === 'noscript' || tagName === 'noembed' || tagName === 'noframes') && regExpTest(EXPRESSIONS.FALLBACK_TAG_CLOSE, currentNode.innerHTML) ) { _forceRemove(currentNode); return true; } ``` - most importantly for this report, the `afterSanitizeElements` hook dispatch at `src/purify.ts`: line 1850~1851. ```tsx /* Execute a hook if present */ _executeHooks(hooks.afterSanitizeElements, currentNode, null); ``` In other words, a normal allowlisted element continues through `_sanitizeElements()` and reaches `hooks.afterSanitizeElements`, but a disallowed-by-default element that is revived by the `CUSTOM_ELEMENT_HANDLING.tagNameCheck` path does not. This creates a policy inconsistency: an application that relies on `afterSanitizeElements` to remove an attribute from all elements will observe that the policy is applied to normal elements but not to custom elements allowed through `CUSTOM_ELEMENT_HANDLING`. In the PoC, the application hook removes `data-bio` from ordinary elements, but the same attribute remains on `<x-bio>` because the custom-element keep path bypasses `afterSanitizeElements`. The attribute itself is inert at sanitize time and DOMPurify is not directly allowing executable SVG/HTML through. The security impact appears when the application-defined custom element later reads the preserved `data-bio` value in `connectedCallback()` and writes it to `innerHTML`, turning the preserved attribute into a second-order XSS gadget. ##### PoC Reproduced on DOMPurify 3.4.11. ##### Steps 1. Save the following HTML to a file, for example `poc.html`. 2. Open it in a browser. 3. Observe that the `div` control loses `data-bio`, while the allowed custom element keeps it. 4. Observe that after `connectedCallback()` runs, the candidate payload is reinserted into the DOM and executes through the custom element’s own sink. ##### HTML PoC ```html <!DOCTYPE html> <html> <head> <meta charset="UTF-8"> <script src="https://cdnjs.cloudflare.com/ajax/libs/dompurify/3.4.11/purify.min.js"></script> </head> <body> <pre id="result"></pre> <script> window.__controlFired = false; window.__candidateFired = false; customElements.define("x-bio", class extends HTMLElement { connectedCallback() { const bio = this.getAttribute("data-bio"); if (bio) this.innerHTML = bio; } }); DOMPurify.addHook("afterSanitizeElements", node => { if (node.hasAttribute && node.hasAttribute("data-bio")) { node.removeAttribute("data-bio"); } }); const config = { CUSTOM_ELEMENT_HANDLING: { tagNameCheck: /^x-/ } }; const controlInput = '<div data-bio="<img src=x onerror=window.__controlFired=true>"></div>'; const candidateInput = '<x-bio data-bio="<img src=x onerror=window.__candidateFired=true>"></x-bio>'; const cleanControl = DOMPurify.sanitize(controlInput, config); const cleanCandidate = DOMPurify.sanitize(candidateInput, config); const container = document.createElement("div"); container.innerHTML = cleanCandidate; document.body.appendChild(container); setTimeout(() => { document.getElementById("result").textContent = "This is not direct DOMPurify XSS.\n" + "The payload becomes executable only after x-bio writes data-bio into innerHTML.\n\n" + "control: " + cleanControl + "\n" + "candidate: " + cleanCandidate + "\n" + "after connectedCallback: " + container.innerHTML + "\n" + "control fired: " + window.__controlFired + "\n" + "candidate fired: " + window.__candidateFired; }, 100); </script> </body> </html> ``` ##### Expected result ``` control: <div></div> candidate: <x-bio data-bio="<img src=x onerror=window.__candidateFired=true>"></x-bio> after connectedCallback: <x-bio data-bio="..."><img src="x" onerror="window.__candidateFired=true"></x-bio> control fired: false candidate fired: true ``` This is output of HTML PoC. <img width="1917" height="961" alt="poc" src="https://github.com/user-attachments/assets/80e22989-5779-42f8-8ffb-106e9a4c2b10" /> ##### Impact This does not appear to affect DOMPurify’s default configuration as a direct sanitizer bypass. The impact is limited to applications that: - enable `CUSTOM_ELEMENT_HANDLING`, - rely on `afterSanitizeElements` as a security policy layer, - expect that hook to apply uniformly to all surviving elements, - and have allowed custom elements that later re-inject preserved attribute values into `innerHTML` or another HTML sink. In that situation, the behavior can become a second-order XSS gadget because a security-relevant attribute is removed from normal elements but remains on allowed custom elements. Possible fixes or mitigations might include - ensuring that allowed custom elements also consistently pass through `afterSanitizeElements` - documenting clearly that elements preserved via `CUSTOM_ELEMENT_HANDLING` may not participate in the same post-element hook flow as normal allowlisted elements. #### Severity - CVSS Score: 2.1 / 10 (Low) - Vector String: `CVSS:4.0/AV:N/AC:H/AT:N/PR:N/UI:A/VC:N/VI:N/VA:N/SC:L/SI:L/SA:N` #### References - [https://github.com/cure53/DOMPurify/security/advisories/GHSA-c2j3-45gr-mqc4](https://redirect.github.com/cure53/DOMPurify/security/advisories/GHSA-c2j3-45gr-mqc4) - [https://github.com/cure53/DOMPurify/pull/1537](https://redirect.github.com/cure53/DOMPurify/pull/1537) - [https://github.com/cure53/DOMPurify/commit/a9ca1e537422319a557a9a2aa61f003b23b4a197](https://redirect.github.com/cure53/DOMPurify/commit/a9ca1e537422319a557a9a2aa61f003b23b4a197) - [https://github.com/cure53/DOMPurify](https://redirect.github.com/cure53/DOMPurify) - [https://github.com/cure53/DOMPurify/releases/tag/3.4.12](https://redirect.github.com/cure53/DOMPurify/releases/tag/3.4.12) This data is provided by [OSV](https://osv.dev/vulnerability/GHSA-c2j3-45gr-mqc4) and the [GitHub Advisory Database](https://redirect.github.com/github/advisory-database) ([CC-BY 4.0](https://redirect.github.com/github/advisory-database/blob/main/LICENSE.md)). </details> --- ### Release Notes <details> <summary>cure53/DOMPurify (dompurify)</summary> ### [`v3.4.12`](https://redirect.github.com/cure53/DOMPurify/releases/tag/3.4.12): DOMPurify 3.4.12 [Compare Source](https://redirect.github.com/cure53/DOMPurify/compare/3.4.11...3.4.12) - Fixed an issue where a hook would not get called for custom elements, thanks [@​Rikuxx0](https://redirect.github.com/Rikuxx0) - Hardened the handling of hooks removing elements, [@​mkrause-bee360](https://redirect.github.com/mkrause-bee360) - Added support for a few new SVG attributes, thanks [@​cbn-falias](https://redirect.github.com/cbn-falias) & [@​Develop-KIM](https://redirect.github.com/Develop-KIM) - Hardened the handling of declarative partial updates - Updated the documentation is several spots, README, wiki, etc. - Bumped several dependencies where possible </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/kenn-io/agentsview). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yNzIuNCIsInVwZGF0ZWRJblZlciI6IjQzLjI3Mi40IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119--> Co-authored-by: renovate[bot] <renovate[bot]@users.noreply.github.com>
Some OpenAI-compatible constrained-decoding servers expand JSON Schema string-length bounds into grammar productions. The recall body limit of 5,000 characters can make that generated grammar too large for the server to parse, causing every extraction request to fail before inference even though the endpoint supports `json_schema`. Keep the 5,000-character body limit in AgentsView's existing client-side response validation, while omitting only that large bound from the model-facing schema. Entry counts, title lengths, entity counts, and entity lengths remain constrained in both places, and the transport cap plus local validation still prevent oversized responses from reaching the archive. The response reader consumes one sentinel byte beyond that cap so oversized successful output is identified explicitly instead of becoming a transient truncated-JSON failure. Because these limits are client-only, an oversized generated body or successful transport response fails its source session behind the normal backoff while the pass continues through later candidates. Actual violations of constraints still present in the request schema remain endpoint-scoped and abort the pass. The extraction protocol version advances so output produced under the compatible request contract receives a new generation fingerprint instead of mixing with an older corpus. Co-authored-by: Wes McKinney <wesm@users.noreply.github.com>
Configuration is foundational setup information, but its navigation link sat below feature-specific and advanced guides. This moves it directly below Quick Start so readers can find setup details before following guides that depend on them. Co-authored-by: Wes McKinney <wesm@users.noreply.github.com>
Large recall candidate sets could exceed SQLite’s bind-variable limit when evidence was hydrated in one query, causing otherwise valid recall queries to fail. Evidence hydration now uses the existing bounded query helper and accumulates each chunk into the same result map. This preserves evidence ordering within each entry while bounding the number of variables in every query. The regression covers a candidate set above SQLite’s default bind limit with evidence at both ends. Co-authored-by: Wes McKinney <wesm@users.noreply.github.com>
…-io#1219) kenn-io#1202 was closed because being dumb when sync my repo 🤦 Co-authored-by: TzeKei Lee <chikei@users.noreply.github.com>
The seed-and-refresh loop in cmd/agentsview previously only talked to the LiteLLM pricing catalog. When the upstream fetch failed (offline, DNS broken, rate-limited) and the embedded fallback snapshot did not contain the user's model, daily usage cost silently dropped to $0 — the same symptom that the fork model custom-pricing test guards against. Wire in OpenRouter's public /models endpoint as a second background source. LiteLLM stays first because it covers the public models agentsview normally parses; OpenRouter fills in fork-tuned and private model prices LiteLLM has not yet picked up. Each fetch failure is logged but never aborts the loop, so a partial outage of one upstream does not prevent the other from seeding. All successful results are merged with first-non-zero precedence per model_pattern. Adds: - internal/pricing/catalog/openrouter.go: fetcher and parser - internal/pricing/openrouter_test.go: unit tests for parser filtering, per-token-to-per-million conversion, and merge precedence - DefaultPricingSources() and MergePricing() in litellm.go exposing the source list and the merge helper so other callers (CLI statusline, future config-driven sources) can reuse them - refreshPricingFromSources() in cmd/agentsview/usage.go replacing the previous single-source call Also bumps the default-port assertion in cmd/agentsview/main_test.go and pg_test.go from 8080 to 9765 to match the port change landed earlier.
…aths
Add two regression tests around GetDailyUsage so future changes
that silently drop unpriced models can be caught immediately.
TestGetDailyUsageForkModelPricing inserts a custom model pattern
("internal-private-model") via UpsertModelPricing and verifies
that the day entry exists with the expected input/output token
counts and a non-zero cost. This is the case the user hit in
June 2026: a downstream fork that uses internal/private model
identifiers saw $0.00 cost because the model name did not
canonicalize to any upstream LiteLLM catalog key, even though
the database had rows with valid token_usage payloads.
TestGetDailyUsageUnknownModelHasZeroCostButCountsTokens covers
the fall-through case where a model is genuinely not priced:
the day entry must still exist (tokens are still counted), but
cost is $0. If this test ever fails with len(Daily)==0 the
upstream time-window SQL from issue kenn-io#904 has regressed.
seedPricing already kicks off one LiteLLM + OpenRouter fetch at startup and reapplies custom_model_pricing on top. But newly released or repriced upstream models were never picked up without a restart, which meant the dashboard could show stale rates for weeks between agentsview upgrades. periodicPricingRefresh runs a ticker in the server goroutine for the whole lifetime of the process. Every tick it reruns refreshPricingFromSources (LiteLLM merged with OpenRouter) and reapplies cfg.CustomModelPricing so a newly-published upstream rate cannot silently shadow the user's own override for fork/private model names. The loop is a no-op when interval <= 0 and unwinds on context cancel. Interval defaults to 24h — long enough to be gentle on the upstream catalogs, short enough that a mid-week price drop is picked up the next day.
The previous filter required strict text->text modality, which dropped multimodal-input, text-output models like MiniMax-M3 (text+image+video->text) and kimi-k2.5 (text+image->text) — the very models users reach via bare names such as `MiniMax-M3`. They still bill prompt/completion in text tokens, so filter on the output side only. Combined with the unqualified-suffix alias, sessions that log bare model names now resolve pricing directly from OpenRouter's public catalog.
OpenRouter ids are provider-qualified (`minimax/minimax-m3`), but agentsview sessions frequently record bare model names (`MiniMax-M3`, `kimi-k2.5`). The canonical resolver refused those lookups because every candidate key had a provider prefix and the same rank, so multiple providers tied and the row stayed unpriced. When a bare suffix is unique across the OpenRouter catalog, also emit an unqualified ModelPricing row so the resolver can rank it at the unqualified tier and resolve a bare user-side model. Shared suffixes (two providers publishing `kimi-k2.5`) still only produce prefixed rows to avoid fabricating OpenRouter-internal ambiguity.
cmd/agentsview/seedPricing writes the LiteLLM fallback snapshot to model_pricing and fires a background multi-source refresh, but it never wired the config-driven [custom_model_pricing] map into the running DB. The CLI statusline and pg serve paths call applyCustomPricing explicitly, which masked the gap, but the embedded HTTP server left db.customPricing at its zero value and silently priced fork-private models at \$0. Call applyCustomPricing immediately after seedPricing so fork owners can configure their internal-model rates in config.toml and have them flow into every GetDailyUsage call without a CLI detour. The read path was already correct: loadPricingMap merges db.customPricing on top of whatever model_pricing returns, so this fix is purely about plumbing the writer.
Cherry-pick bookkeeping for the upcoming upstream PR: - main_test.go, pg_test.go: restore 8080 default (the c876288 fork commit bundled in a 9765 port change that is not part of this PR) - usage_test.go: drop the TestRefreshPricingIfStale_* / TestEnsurePricingWithFetcher* tests that reference fork-local helpers (refreshPricingIfStale, ensurePricingWithFetcher) that do not exist on upstream; the upstream equivalent lives in the pricingrefresh package and is tested there - usage.go: add a local upsertPricing helper used by seedFallbackPricing and refreshPricingFromSources. The same helper exists inside internal/pricingrefresh on upstream but is unexported, so we duplicate the few lines here to keep the PR self-contained - openrouter.go, litellm.go: gofmt -s alignment cleanup - main.go: drop a stray seedPricing line left over from the 79f862b conflict resolution (applyCustomPricing is the only startup hook on this path on upstream) No functional change to the OpenRouter fetcher, the 24h refresh loop, or the parser updates.
mjacobs
pushed a commit
that referenced
this pull request
Jul 24, 2026
Restore synchronous startup fallback seeding while keeping the initial network refresh asynchronous, and remove the periodic custom-pricing map write that raced with request reads. Track OpenRouter aliases so refreshes can remove obsolete bare names locally and in PostgreSQL, preserve ordered LiteLLM precedence, and accept free-model zero prices. VALID (fixed): #1, #2, #3, kenn-io#4, kenn-io#5 INVALID (dismissed): none PEDANTIC (skipped): none
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds OpenRouter's public /models endpoint as a second background pricing source and wires a 24h refresh loop in the embedded server, so newly-released or repriced models pick up rates without an agentsview restart.
What it does
Why
The seed-and-refresh loop previously only talked to the LiteLLM catalog. When that fetch failed (offline, DNS broken, rate-limited) and the embedded fallback snapshot did not contain the user's model, daily usage cost silently dropped to $0 — the same symptom that the fork-model custom-pricing test guards against. OpenRouter's /models endpoint frequently lists fork-tuned and private model prices LiteLLM has not yet picked up, so a partial outage of one upstream should not prevent the other from seeding. The 24h loop covers the case where a model is repriced or newly published between agentsview upgrades.
Out of scope (intentionally not in this PR)