Skip to content

feat(server): loopback control API + per-batch progress + idle-exit (#30, PR-2a)#38

Merged
kfastov merged 2 commits into
mainfrom
feat/issue-30-backfill-server-2a
Jun 3, 2026
Merged

feat(server): loopback control API + per-batch progress + idle-exit (#30, PR-2a)#38
kfastov merged 2 commits into
mainfrom
feat/issue-30-backfill-server-2a

Conversation

@kfastov

@kfastov kfastov commented Jun 2, 2026

Copy link
Copy Markdown
Owner

PR-2a of #30 — server-side foundation for server-executed backfill. The CLI client (backfill --chat X foreground/background, backfill status/count/wait/cancel, auto-start) consumes this in PR-2b.

A. Always-on loopback control API

  • New core/control-server.js + a listener in mcp-server.js, bound to 127.0.0.1 (config control.{enabled,host,port}, default true/127.0.0.1/8765), independent of mcp.enabled. The existing MCP listener is untouched.
  • Auth: a per-store 192-bit token written to <store>/control.json (mode 0600); every /control/* request must send header x-tgcli-control-token, else 401.
  • Endpoints (delegate to the same MessageSyncService methods the MCP tools use — no duplicated job logic):
    • GET /control/ping{ ok, pid, version, startedAt, jobs: { pending, inProgress } }
    • POST /control/backfill {chatId, depth?, minDate?}addJob + processQueue{ jobId, channelId, status }
    • POST /control/cancel {chatId?|jobId?}{ canceled, jobIds }

B. Per-batch progress persistence

  • New started_at jobs column (via the idempotent _ensureJobColumn); _backfillHistory now updates message_count/cursor/updated_at (and stamps started_at on the first batch) every batch, so backfill status/count (PR-2b) reflect live progress. New helpers getJobCounts() and getWatchedChannelCount().

C. Idle-exit

  • server --idle-exit <duration> (forwarded to the worker child). Idle = 0 pending + 0 in-progress jobs AND 0 watched channels (sync_enabled=1) AND control API quiet longer than the window → graceful shutdown (removes control.json). No flag → always-on. So any active channels watch keeps the daemon alive (it needs realtime).

Why

The server is a lock-free WAL writer and CLI reads coexist with it. Routing backfill through it (PR-2b) means the CLI won't take its own exclusive write lock → single writer, no lock contention.

Scope — not here

Server-side only: part of #30, not a close. PR-2b adds the CLI backfill client + auto-start; PR-3 routes realtime/auth through the single writer. The MCP listener is unchanged.

Tests

npm test252 passing (224 baseline + 28 new across 4 files): control endpoints incl. 401 on missing/wrong token; per-batch progress with a temp DB + stub client; idle predicate truth table + fake-timer shutdown + control.json 0600 lifecycle; control.* config normalization. No real network.

Noted hardening follow-ups (negligible on a loopback+token API; not blocking)

  • Token compare is plain === — could use crypto.timingSafeEqual.
  • No request-body size cap on the control endpoints.

kfastov added 2 commits June 2, 2026 17:32
…#30)

Server-side groundwork for the backfill rework (PR-2a). The CLI backfill
client commands land separately in PR-2b.

Part A — Always-on loopback control listener (mcp-server.js,
core/control-server.js): a dedicated HTTP server bound to 127.0.0.1 that
runs whenever the server runs, independent of mcp.enabled. The existing
MCP listener is untouched. New config keys control.enabled (default true),
control.host (127.0.0.1), control.port (8765) follow the mcp.* convention.
On startup the server writes <store>/control.json (mode 0600) with
{ pid, port, token, startedAt, version } and removes it on graceful
shutdown. Every /control/* request must carry x-tgcli-control-token or
gets 401. Endpoints delegate to the same MessageSyncService methods the
MCP tools use: GET /control/ping, POST /control/backfill (addJob +
processQueue, same path as scheduleMessageSync), POST /control/cancel
(same logic as `sync jobs cancel`). Each authed request bumps
lastControlActivityAt for the idle monitor.

Part B — Per-batch progress persistence (message-sync-service.js): adds a
started_at jobs column via the idempotent _ensureJobColumn helper.
_backfillHistory now writes message_count, cursor, updated_at (and
started_at on the first batch) once per batch via _updateJobProgress,
keeping terminal updates intact. Adds getJobCounts() and
getWatchedChannelCount() helpers.

Part C — Idle-exit (mcp-server.js, cli.js): an idle monitor active only
when --idle-exit is configured. Idle = 0 pending + 0 in-progress jobs AND
0 watched channels AND control API quiet longer than the window; it then
shuts down gracefully. cli.js `server --idle-exit <duration>` parses with
parseDuration and passes the value to the spawned mcp-server.js child via
argv; without it the server stays up forever.
…count wrapper (#30)

Address audit findings on PR-2a (behavior unchanged, suite green).

1. Shared HTTP body reader: extract the character-identical readJsonBody (plus
   the sendJson helper) into a new core/http-util.js. core/control-server.js now
   imports both; mcp-server.js's local readBody is replaced by the shared
   readJsonBody at its single call site. Other inline res.writeHead().end()
   sites in mcp-server.js are intentionally left as-is.

2. Shared duration parser: extract parseDuration into core/duration.js (returns
   ms; a bare number means seconds, matching the long-standing CLI behavior).
   cli.js imports it (local copy removed; all call sites unchanged).
   mcp-server.js's parseIdleExitMs (which re-implemented the parser with a
   bare-number-means-ms divergence) is replaced by resolveIdleExitMs: a plain
   integer (forwarded by the CLI, already in ms) passes through as ms, while a
   unit-suffixed string (e.g. from TGCLI_IDLE_EXIT) goes through the shared
   parseDuration. The ms-vs-string split is documented at the call site so the
   divergence is explicit, not accidental.

3. getJobCounts wrapper: reduced to a trivial one-line re-key of getQueueStats()
   rather than removed. Keeping the clean { pending, inProgress } contract keeps
   the /control/ping handler, the idle monitor, and their minimal service stubs
   off the snake_case queue shape; inlining getQueueStats() at those call sites
   would have coupled them (and the tests) to in_progress.

Adds tests/duration.test.js for the newly shared parseDuration.
@kfastov kfastov force-pushed the feat/issue-30-backfill-server-2a branch from cd64313 to fa8d1a7 Compare June 3, 2026 09:10
@kfastov kfastov merged commit bfba97f into main Jun 3, 2026
@kfastov kfastov deleted the feat/issue-30-backfill-server-2a branch June 3, 2026 09:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant