perf+refactor: DB hot-path index, dedup, classifier split, history bound - #63
Merged
Conversation
Pher217
force-pushed
the
claude/2026-06-15-perf-refactor
branch
from
June 15, 2026 21:04
85eb3b3 to
f90df11
Compare
These two gate CI as blocking steps; with unpinned `>=` floors a new release silently turns CI red with no code change (which is how the lint/ security steps drifted red on main). Pin to the current known-good minor; mypy stays unpinned since its CI step is advisory.
- Add a DB index on Thread.external_session_ref — it is filtered on every observed line (observe/service.py) and every host WS frame (hostlink/consumers.py) and was previously a sequential scan. - Replace the per-turn MAX(sequence) aggregate with an indexed order_by(-sequence).first() in observe/service.py and threads/dispatch.py. - Observer loop: log scan failures with full traceback + back off after 5 consecutive errors (was a silent one-line stderr write). - Bound the per-runtime dedup 'seen' set (was unbounded for process lifetime). - Log the swallowed exception before each HTML->plain Telegram fallback. No behavior change. backend 556 pass / 1 pre-existing fail.
…_esc
- New apps/core/ utility package (no models, not in INSTALLED_APPS):
- BearerTokenPermission base — connectors + gateway permission gates now
subclass it (settings name + request-flag parameterized). connectors keeps
its Ed25519 short-circuit, then delegates the legacy bearer path to super().
- html._esc — shared by observe/formatting and slash/handlers/sessions.
- observe/runtimes: JsonlScanMixin provides the one scan_file_meta shared by the
four JSONL adapters; _cwd_to_repo() replaces the repeated cwd->repo expression.
opencode (sqlite) untouched; each adapter keeps its own parse logic.
Behavior-identical refactor. 166 targeted tests + full suite green (556/1 pre-existing).
_build_history loaded every message of a thread into the LLM context on every turn (unbounded DB read + context growth). Cap at the 200 most recent allowed-role messages, returned in chronological order. Identical output for threads with <=200 messages; longer threads keep the most recent window. Adds a test asserting the bound, ordering, and oldest-dropped behavior. Behavior change for long threads (older context is dropped) — deliberate.
Pher217
force-pushed
the
claude/2026-06-15-perf-refactor
branch
from
June 15, 2026 21:12
f90df11 to
2e69881
Compare
Pher217
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second cleanup pass (after #59, now merged). Rebased onto current
main; mergeable / conflict-free. Implemented by Sonnet + Kimi k2.7 workers under review; I reviewed every diff, ran tests, committed. Independently reviewed by Codex (gpt-5.5) — no blocker/major behavior-equivalence problems.6 commits
Thread.external_session_ref(was a seq scan on every observed line/host frame);MAX(sequence)→indexedorder_by(-sequence).first(); observer logs failures w/ traceback + backoff; bounds the dedupseenset; logs swallowed HTML→plain fallbacks.apps/core/auth.py::BearerTokenPermission(connectors+gateway),apps/core/html.py::_esc,JsonlScanMixin+_cwd_to_repofor the 4 JSONL adapters. Behavior-identical (Codex-confirmed: settings, flags,compare_digest, Ed25519 short-circuit unchanged).detect_major_steps→_classify_against_prior+_finished_by_disappearance. Behavior-identical._build_historyto the 200 most recent messages.Thread as _T).❗ Dropped from this PR: the
telegram/service.pysplitI split that 636-line file into modules earlier, but
telegram/service.pyis the single hottest file in the repo — it collided with every telegram PR merged this session (#58, #60, #62, #64). Keeping the split here is a perpetual rebase treadmill. The split work is preserved on branchperf-refactor-with-telegram-split(Codex-reviewed, all tests green) and can be reapplied in a quiet window once telegram dev settles. This PR keeps the 6 conflict-free, high-value commits.Test summary (fresh, local, on current main)
git diff --checkclean · 575 pass / 3 skip / 1 fail (pre-existingtest_chat_surface_thread_excluded_from_fleet).test_wsclient.py); my host-agent code is byte-identical tomain.Supervisor notes — proposed items deliberately NOT done
(provider,label)unique constraint — a worker added it; reverted (public ModelViewSet → dup POST would 500; migration could fail on existing dupes).GinIndex(metadata)won't help the->>equality lookup; needs an expression index or column migration).CancelledError; changing async shutdown semantics risks reliability per fix(host-agent): daemon heartbeat + watchdog to recover stale channel subscription #56/fix(host-agent): reliable TUI submit — separate Enter, sent twice #57).