feat(server): loopback control API + per-batch progress + idle-exit (#30, PR-2a)#38
Merged
Merged
Conversation
…#30) Server-side groundwork for the backfill rework (PR-2a). The CLI backfill client commands land separately in PR-2b. Part A — Always-on loopback control listener (mcp-server.js, core/control-server.js): a dedicated HTTP server bound to 127.0.0.1 that runs whenever the server runs, independent of mcp.enabled. The existing MCP listener is untouched. New config keys control.enabled (default true), control.host (127.0.0.1), control.port (8765) follow the mcp.* convention. On startup the server writes <store>/control.json (mode 0600) with { pid, port, token, startedAt, version } and removes it on graceful shutdown. Every /control/* request must carry x-tgcli-control-token or gets 401. Endpoints delegate to the same MessageSyncService methods the MCP tools use: GET /control/ping, POST /control/backfill (addJob + processQueue, same path as scheduleMessageSync), POST /control/cancel (same logic as `sync jobs cancel`). Each authed request bumps lastControlActivityAt for the idle monitor. Part B — Per-batch progress persistence (message-sync-service.js): adds a started_at jobs column via the idempotent _ensureJobColumn helper. _backfillHistory now writes message_count, cursor, updated_at (and started_at on the first batch) once per batch via _updateJobProgress, keeping terminal updates intact. Adds getJobCounts() and getWatchedChannelCount() helpers. Part C — Idle-exit (mcp-server.js, cli.js): an idle monitor active only when --idle-exit is configured. Idle = 0 pending + 0 in-progress jobs AND 0 watched channels AND control API quiet longer than the window; it then shuts down gracefully. cli.js `server --idle-exit <duration>` parses with parseDuration and passes the value to the spawned mcp-server.js child via argv; without it the server stays up forever.
…count wrapper (#30) Address audit findings on PR-2a (behavior unchanged, suite green). 1. Shared HTTP body reader: extract the character-identical readJsonBody (plus the sendJson helper) into a new core/http-util.js. core/control-server.js now imports both; mcp-server.js's local readBody is replaced by the shared readJsonBody at its single call site. Other inline res.writeHead().end() sites in mcp-server.js are intentionally left as-is. 2. Shared duration parser: extract parseDuration into core/duration.js (returns ms; a bare number means seconds, matching the long-standing CLI behavior). cli.js imports it (local copy removed; all call sites unchanged). mcp-server.js's parseIdleExitMs (which re-implemented the parser with a bare-number-means-ms divergence) is replaced by resolveIdleExitMs: a plain integer (forwarded by the CLI, already in ms) passes through as ms, while a unit-suffixed string (e.g. from TGCLI_IDLE_EXIT) goes through the shared parseDuration. The ms-vs-string split is documented at the call site so the divergence is explicit, not accidental. 3. getJobCounts wrapper: reduced to a trivial one-line re-key of getQueueStats() rather than removed. Keeping the clean { pending, inProgress } contract keeps the /control/ping handler, the idle monitor, and their minimal service stubs off the snake_case queue shape; inlining getQueueStats() at those call sites would have coupled them (and the tests) to in_progress. Adds tests/duration.test.js for the newly shared parseDuration.
cd64313 to
fa8d1a7
Compare
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR-2a of #30 — server-side foundation for server-executed backfill. The CLI client (
backfill --chat Xforeground/background,backfill status/count/wait/cancel, auto-start) consumes this in PR-2b.A. Always-on loopback control API
core/control-server.js+ a listener inmcp-server.js, bound to 127.0.0.1 (configcontrol.{enabled,host,port}, defaulttrue/127.0.0.1/8765), independent ofmcp.enabled. The existing MCP listener is untouched.<store>/control.json(mode0600); every/control/*request must send headerx-tgcli-control-token, else401.MessageSyncServicemethods the MCP tools use — no duplicated job logic):GET /control/ping→{ ok, pid, version, startedAt, jobs: { pending, inProgress } }POST /control/backfill {chatId, depth?, minDate?}→addJob+processQueue→{ jobId, channelId, status }POST /control/cancel {chatId?|jobId?}→{ canceled, jobIds }B. Per-batch progress persistence
started_atjobs column (via the idempotent_ensureJobColumn);_backfillHistorynow updatesmessage_count/cursor/updated_at(and stampsstarted_aton the first batch) every batch, sobackfill status/count(PR-2b) reflect live progress. New helpersgetJobCounts()andgetWatchedChannelCount().C. Idle-exit
server --idle-exit <duration>(forwarded to the worker child). Idle = 0 pending + 0 in-progress jobs AND 0 watched channels (sync_enabled=1) AND control API quiet longer than the window → graceful shutdown (removescontrol.json). No flag → always-on. So any activechannels watchkeeps the daemon alive (it needs realtime).Why
The server is a lock-free WAL writer and CLI reads coexist with it. Routing backfill through it (PR-2b) means the CLI won't take its own exclusive write lock → single writer, no lock contention.
Scope — not here
Server-side only: part of #30, not a close. PR-2b adds the CLI
backfillclient + auto-start; PR-3 routes realtime/auth through the single writer. The MCP listener is unchanged.Tests
npm test→ 252 passing (224 baseline + 28 new across 4 files): control endpoints incl.401on missing/wrong token; per-batch progress with a temp DB + stub client; idle predicate truth table + fake-timer shutdown +control.json0600 lifecycle;control.*config normalization. No real network.Noted hardening follow-ups (negligible on a loopback+token API; not blocking)
===— could usecrypto.timingSafeEqual.