Graph projection is O(entire history) with full row materialization — needs incremental projection + streaming reads (prod graph frozen 3 weeks)

[bgmcmullen]

Summary

Graph projection is a full-history rebuild with full materialization: every run executes SELECT <cols> FROM <sourceDataset> — no WHERE, no watermark (plugins-workspace/context-graph/src/project.js, the LLP 0096 shared scan) — through executeQuerySql, which collects every row into a JS array (result.rows) before the rule loop starts. Both the run's memory and its time are O(entire history), so the design has a scaling expiry date independent of any budget or host size. On the hyperparam prod server it has already expired: the kernel's 1 GiB heap-growth budget (LLP 0097) now refuses every projection run — correctly, since millions of materialized rows are genuine retention — and the graph has been frozen at 2026-07-10 for three weeks (hypaware-server#143).

The output side is NOT the problem: three months of fleet history projects to only 9,904 nodes / 28,151 edges — the node/edge maps are trivial. It's the input rows that are materialized wholesale.

Why each cheap fix expires

• Raise the budget for projection runs: retained rows grow linearly with ingest (hyperparam prod: 202k rows on 07-10 → 1.66M+ by 07-22). Any ceiling is walked to within weeks-to-months, and past it lies the OOM class the budget exists to prevent. Band-aid only.
• Slice the scan by date partition: bounds memory durably (peak retention = one slice's rows; the accumulated node/edge maps stay small). But each run still scans all of history — a full pass costs 17–42s today, grows linearly forever, and runs daily on the same event loop that serves queries and ingest.

Proposed

1. Incremental projection with a per-scope watermark: each automatic run reads only rows past the last successful run's high-water mark (date partition + a stable tiebreaker). The graph's upsert/first_seen semantics already accommodate append-style updates. O(new data) per run, constant-ish forever.
2. Streaming or sliced reads for internal callers: projection should iterate rows without materializing the full result — either a streaming interface on executeQuerySql for trusted daemon jobs, or per-date-slice queries with rules applied per slice. This keeps full REBUILDS possible at any history size.
3. Full rebuild becomes the recovery path, not the daily path: kept for contract changes, run sliced (and ideally admin-triggered/offline), preserving the derived-rebuildable philosophy (LLP 0053) without paying its cost daily.

Context

Found while diagnosing hypaware-server#143 (prod graph frozen at 07-10 across the 07-16 redeploy that shipped the heap budget): the issue-#9 OOM fix and automatic projection (#124) are incompatible as shipped, and the freeze self-perpetuates (scheduler retries daily into the same refusal). The band-aid/slice/watermark ladder is written up there; this issue tracks the durable fix.

[philcunliffe]

🤖 neutral: the diagnosis holds against the tree, but the durable fix is a design decision, not a bugfix

I verified the finding rather than taking it on trust, and it is accurate. I am
declining to open a fix PR and recording why: what this issue asks for is an
architectural change that needs a design owner, and two of its three proposals are
entangled with work that is already in flight and already stuck.

What I confirmed in the tree

hypaware-core/plugins-workspace/context-graph/src/project.js at master:

• No WHERE, no watermark. Line 100: `SELECT ${[...columns].sort().join(', ')} FROM ${contract.sourceDataset}`, and line 205 for the id scan. Neither carries a predicate.
• Full materialization before the rule loop. Lines 107/109 and 137/139 read result.rows.length and then for (const row of result.rows), so the whole result set is a JS array before any rule runs.

So both halves of the diagnosis (O(entire history) in time and in peak memory) are
properties of the code as written, not an inference from prod symptoms.

Why I am not writing the fix

1. The watermark is blocked on an ordering decision this repo has deliberately not made.
Proposal 1 needs a high-water mark of "date partition + a stable tiebreaker". The
projection's own documentation says the opposite is true today, at project.js:296-307:

"equal (or unknown) times fall back to a value comparison so the result is independent
of merge order, which matters because the projection's source SELECTs have no stable
ordering"

That comment carries @ref LLP 0023#merge-policy [implements]. Order-independence is not
an accident here, it is an implemented decision, and mergeRow is shared with the dedup
compaction in maintenance.js specifically so projection-time and compaction-time merges
agree. Introducing a stable tiebreaker to hang a watermark on is therefore a change to
LLP 0023's merge policy and to a contract two code paths share. That is a decision to be
recorded in an LLP, not one to make inside a fix/issue-489 branch.

2. Streaming reads collide with a change set that is in flight and stuck.
Proposal 2 (a streaming interface on executeQuerySql for trusted daemon jobs) is
substantially LLP 0055, "Stream Aggregates via scanColumn Rather Than Buffering
Rows" - part of the bounded-query-execution change set (LLP 0054/0055/0056/0057/0058/0059),
which is open as PR #221 and currently labelled neutral:stuck. #221's own body
records that it is blocked on two unpublished upstream releases:

• hyparam/icebird#28 - the scanColumn streaming hook, needed before the pinned icebird@0.8.11 can be moved
• hyparam/squirreling#42 - the execution budget and QueryBudgetExceededError, likewise pinned at squirreling@0.12.24

Writing a second streaming path now would either duplicate that change set or preempt the
design it is waiting on, and would land in the same query plane it is about to rewrite.

3. There is no reproducing test I can write that would mean anything.
Neutral's gate for opening a fix PR is a regression test that fails before the fix and
passes after (LLP 0002). Here the failing condition is "peak retention grows with total
history", and the fix is the feature: a test for incremental projection cannot fail
meaningfully until incremental projection exists. I would be writing the test to fit the
implementation, which is exactly the shape the gate exists to reject.

What I need from you

This is a scope-and-sequencing call, and it is yours. Concretely:

1. Do you want this to enter the pipeline family as a request LLP (spec -> design ->
   plan -> implementation), rather than the maintenance family? That is my recommendation.
   The corpus is where the ordering decision (point 1) and the relationship to LLP 0023,
   0053, 0096, 0097 belong, and a request LLP is a statement of intent I should not invent
   on your behalf. If you author one, the backlog picks it up on the next tick with no
   further action from you.

2. Or do you want a bounded interim fix now? The one rung of your own ladder that is
   not blocked is date-partition slicing (your proposal 2, second half): apply the
   rules per slice so peak retention is one slice rather than all history. It leaves the
   full-history scan cost in place, which your issue correctly calls out as still growing
   linearly, but it is implementable today, needs no upstream release, needs no ordering
   decision, and would unfreeze the prod graph. It is testable as written (project a
   synthetic multi-partition dataset, assert peak retention tracks one slice). Say the
   word and I will take just that.

3. Or is unfreezing prod urgent enough that you would rather raise the LLP 0097
   heap-growth budget for projection runs as an explicit, time-boxed band-aid? Your issue
   argues against this and I agree with the argument, so I will not do it unattended, but
   if the three-week freeze is costing more than the risk, that is your trade to make.

I have not touched project.js, opened a branch, or changed anything.

How to unstick

Reply with a comment on this issue. Neutral monitors this thread and will re-engage
with your guidance on its next tick - naming the option is enough. If you author a request
LLP instead, the backlog reconciler picks it up automatically and you can leave this issue
to close on its own.

Related

• PR Bounded query execution: /v1/query OOM fix (LLP 0054-0059) [blocked by upstream] #221 (bounded-query-execution, neutral:stuck), LLP 0054/0055 - the streaming work this overlaps
• LLP 0023 #merge-policy - the order-independence the watermark would change
• LLP 0096 (one shared scan per contract), LLP 0097 (heap-growth guard), LLP 0053
• hypaware-server#143 - the prod freeze this tracks

[philcunliffe]

Decision: take option 3 now - raise the LLP 0097 heap-growth budget for projection runs as an explicit, time-boxed band-aid so the prod graph unfreezes. Scope it to projection runs only (not the general query plane), and note in the code/PR that it is temporary cover for this issue.

The durable fix goes through the pipeline family: I will author a request LLP for incremental projection (watermark + streaming/sliced reads), which will settle the LLP 0023 ordering question properly. Leave this issue open to track that; the budget raise alone does not close it.

[philcunliffe]

🤖 neutral: your decision is already on master, so there is nothing for me to build

You decided option 3 - raise the LLP 0097 heap-growth budget for projection runs only, as a
time-boxed band-aid - and I picked the issue up to implement it. I am declining to open a PR
because that change already shipped, five days before the decision, and I would be
re-landing it.

I verified this against origin/master (73b4618) rather than the stale checkout, and every
point below is an executed check, not an inference.

The evidence

PR #378 - "context-graph: dedicated finite projection heap budget
(HYP_GRAPH_PROJECTION_MAX_HEAP_MB) - step 1 of #376" - merged 2026-07-25. On master
today, hypaware-core/plugins-workspace/context-graph/src/project.js already has exactly the
seam your decision describes:

your constraint	state on master
raise the budget for projection runs	resolveProjectionMaxHeapBytes() at :41, reads a dedicated HYP_GRAPH_PROJECTION_MAX_HEAP_MB, default 3 GiB, and deliberately refuses the "0 disables" escape
applied to projection runs	passed as maxHeapBytes at all three scan sites: shared per-contract scan :141, raw-SQL rule scans :172, dedup id reads :251
not the general query plane	DEFAULT_MAX_HEAP_GROWTH_BYTES is still 1 GiB at src/core/query/sql.js:34, and the graph read surface (context-graph/src/query.js:198) correctly still inherits it

projectGraph's only caller is context-graph/src/command.js:42, so there is no entrypoint
that bypasses the dedicated budget.

The reproduce-first gate confirms it. I wrote the regression test the work would need - a
real executeQuerySql projection run with the general budget squeezed, plus the scoping
counter-assertion - and on pristine origin/master the projection half already passes:

ok 1 - a projection run completes even though the general query budget would refuse it

There is no failing-then-passing reproduction to be had. Per LLP 0002 that means no credible
fix, so no PR. I confirmed the provenance instead by running the repo's own
test/plugins/context-graph-projection-budget.test.js across the #378 boundary: pre-#378
(9eb6ba4^) it is 0 pass / 1 fail (the module has no resolveProjectionMaxHeapBytes); on
master it is 3 pass / 0 fail.

Also worth knowing before you author the request LLP

Issue #377 already exists and is open: "Durable graph projection: incremental
(seq-watermark) + chunked (partition-aligned) rebuild - needs an LLP". That is the durable fix
you said you would author a request LLP for. #376 (the parent of the band-aid) is closed.
So this issue may be a duplicate of a #376/#377 pair that is already tracking both halves.

Two things your decision asked for that genuinely are NOT done

#378 landed the code without the LLP work CLAUDE.md requires, so **constraint 2 of your decision

• "note in the code/PR that it is temporary cover" - is unmet in the tree**:

1. LLP 0097 has no record of the projection-scoped budget at all. Grepping all of llp/ for
   HYP_GRAPH_PROJECTION_MAX_HEAP_MB or resolveProjectionMaxHeapBytes returns nothing. A
   reader of LLP 0097 today cannot learn the exemption exists.
2. The code does not say "temporary." project.js:24-48 frames it as "a dedicated budget,
   not an exemption" and names no tracking issue. The @ref LLP 0097 [constrained-by] is
   present and accurate, but nothing marks it time-boxed. context-graph: dedicated finite projection heap budget (HYP_GRAPH_PROJECTION_MAX_HEAP_MB) - step 1 of #376 #378's PR body called it an
   "acknowledged treadmill"; that never reached the tree.

I did not fix these unilaterally, because they implement a decision whose premise I had just
falsified, and because a docs-only change has no behavioural test and so sits outside the
reproduce-first gate by construction. That is your call to make, not mine to assume.

What neutral needs from you

1. Why is prod still frozen? hyparam/hypaware-server#143 is still open (updated
   2026-07-30 21:25) and this issue says the graph is frozen at 07-10, yet the band-aid merged
   07-25. Either the deployed server predates context-graph: dedicated finite projection heap budget (HYP_GRAPH_PROJECTION_MAX_HEAP_MB) - step 1 of #376 #378 (no release is listed after 07-22;
   master is at 1.19.0), which is a deploy action and not a code change - or 3 GiB has
   already been walked past at 1.66M+ rows, which means bumping
   HYP_GRAPH_PROJECTION_MAX_HEAP_MB and prioritising Durable graph projection: incremental (seq-watermark) + chunked (partition-aligned) rebuild - needs an LLP #377. These need different responses and
   I cannot tell which from here.
2. Should this issue be closed as a duplicate of graph projection retains the whole source table: budget-exempting it trades a stale graph for a daemon OOM; needs finite budget now, incremental+chunked projection as the durable fix #376 (fixed) + Durable graph projection: incremental (seq-watermark) + chunked (partition-aligned) rebuild - needs an LLP #377 (open), or retitled to
   track only the durable fix?
3. Do you want the two documentation gaps closed? If so I will land a one-commit PR: an
   LLP 0097 note recording the projection-scoped budget, and a temporary/Refs #377 marker in
   project.js. Say the word and it is done.

How to unstick

Reply with a comment on this issue. Neutral monitors this thread and will re-engage with
your guidance on its next tick - answering any one of the three above is enough to move it.

Related

• PR context-graph: dedicated finite projection heap budget (HYP_GRAPH_PROJECTION_MAX_HEAP_MB) - step 1 of #376 #378 (the band-aid, merged), issue graph projection retains the whole source table: budget-exempting it trades a stale graph for a daemon OOM; needs finite budget now, incremental+chunked projection as the durable fix #376 (closed), issue Durable graph projection: incremental (seq-watermark) + chunked (partition-aligned) rebuild - needs an LLP #377 (the durable fix, open)
• LLP 0097 (heap-growth guard), LLP 0096, LLP 0023 #merge-policy, hypaware-server#143