[Bug] Cron session-artifact registry only grows; every list() reads all registered partitions

[rain-lee1999]

Summary

AgentCronJobStore in session-artifact mode keeps a map of session id -> scheduled-jobs file. Entries are added from several paths (session bind, store bind, passive catalog, deleted-session cancel). No path removes them. list() reads and parses every registered file on each call. In a long-lived worker with many RLM descendants, the map grows with every session the worker has ever touched, so every list() costs O(sessions ever registered).

The same design caused a real stall on the 0.9.8 TypeScript daemon (numbers below). On current Rust main (e375911) I found the same register-only map and read-all list(), and a unit benchmark shows list() cost grows linearly with registered partitions (see Measured on Rust main). I have not reproduced the stall end to end on Rust.

Observed on 0.9.8 (TS daemon, for context)

One long session, about 1,460 RLM descendants over about 21 h on one worker:

• Per-child idle passivation gap rose from 0-0.9 s to 1.1-4.6 s. Whole sweeps took 35-103 s and then hit request timeouts. Passivation stopped; about 1,040 children were passive and about 400 were still resident.
• 465 idle python kernels (about 2.8 GB RSS) stayed alive because their sessions were never passivated.
• Worker main thread about 99% busy, mostly sync fs (stat/lstat/open/read) plus JSON work. The host ran out of swap (16 GB RAM).
• Supervisor attach, cron_list and heartbeats_list timed out. agent_message.send took minutes.

In TS the store cached parsed state per file, keyed by stat. So each list(), job read or scheduler tick cost one sync statSync per registered artifact, not a re-read. Every passivated descendant stayed registered. The idle sweep called list() once per resident session, which gives O(R x (R + P)) stats per sweep (R = resident, P = passive). Offline, 470 x 1,100 = 517k stats took about 2.05 s per sweep, before swap pressure. Roster flushes and scheduler ticks added O(R + P) each.

A local patch on 0.9.8 removed the growth with three changes: do not register a passive descendant that has no jobs file, unregister after passivation when the file is absent, and compute list() once per sweep.

Root cause on current main

• Register only, no remove: register_session_artifact. No unregister exists in pa-core/src/cron/store.
• Read-all list: list -> read_states runs read_jobs_state (read + serde parse) for every registered file.
• Unlike the TS store, nothing is cached: read_jobs_state runs read_to_string on every call. A missing file costs one failed open; an existing file costs a full read and parse.
• Registration call sites: bind_session, bind_store_artifact, deleted-session cancel.
• A hot caller: the settled-child kernel-release gate calls jobs_store.list() to check one session (worker.rs). It runs on every park of a child worker (worker/turn.rs:123 -> maybe_release_settled_child_kernel), and can_passivate_worker runs it on each idle-eviction timer fire (worker/turn.rs:256). I found no loop that calls list() once per session. The scheduler tick also reads all files twice via mutate_states (state.rs:51,66).

Measured on Rust main (unit benchmark)

An integration test in crates/pa-core/tests builds one AgentCronJobStore::for_session_artifacts() store, registers N partitions with register_session_artifact(), and times list() (median of 7 calls). Case "missing": no partition file exists. Case "present": every file holds {"jobs":[],"dispatches":[]}. The gate column runs N calls of list() plus a filter for one session id, which simulates one release-gate check per child on a shared store. Release build, Apple M3 (arm64, macOS), rustc 1.94.0, warm page cache.

N	list(), missing (ms)	list(), present (ms)	N x list() gate (s)
100	0.25	0.80	0.08
500	0.64	4.06	1.94
1000	1.13	7.66	8.62
2000	2.49	21.38	40.12
5000	11.40	59.58	not run (about 250 s extrapolated)

The jobs lists are empty, so this is a lower bound: real job entries add parse cost.

Reproduction (end to end, not run)

1. Start one worker, then spawn and settle N RLM children (N = 500-1,500) so each gets an artifact partition.
2. Time AgentCronJobStore::list() and the settled-child release gate as N grows.

Expected behavior

• Released or passive sessions are removed from the store, or never registered when they have no jobs file.
• A per-session check like "does this session own an active/paused job" reads only that session's partition, not all of them.

Suggested direction

Any of these would remove the growth:

• Add unregister_session_artifact and call it after passivation, release or delete when the session has no jobs file (this is what fixed the TS case).
• Register a passive or settled session's partition only when its scheduled-jobs file exists.
• Call list() once per sweep or batch instead of once per session, or add a per-session lookup (jobs_for_session) for the release gate.
• Cache parsed states keyed by file mtime/len.

Question for maintainers

Is the long-lived single-worker store expected to accumulate every descendant session? If yes, would a per-session lookup for the release gate be acceptable?

[holny]

Confirmed on current main (74591cd): read_states re-reads and re-parses every registered partition on each call, mutate_states walks all of them twice per tick, and the settled-child release gate's registered_jobs_probe runs a full list() on every park just to answer a question about one session. Register-only map — nothing in cron/store ever removes an entry.

I have a patch underway along the TS lines, since 0.9.8 cached parsed state per file keyed by stat and the re-read looks like a port regression:

• cache parsed state per partition keyed by (mtime, len) — reads cost a stat again, like TS
• answer the release gate from an in-memory active/paused index maintained on the mutate path, keyed by active_session_id (rebinds move jobs across partitions, so the partition key alone won't do)
• skip registering partitions that have no scheduled-jobs file, and drop a registration when its file is gone

Orthogonal to the passivation-gate work in #3840/#3847 — this is about what the jobs probe costs, not when it fires. Branch link to follow on my fork.