Async POST /memories extraction hangs indefinitely on content that dry-run-extract handles in ~30s (0.9.1)

[rschlek]

Async POST /memories extraction hangs indefinitely on content that dry-run-extract handles in ~30s

Version: 0.9.1-slim
Deployment: single-node Docker; PostgreSQL; embeddings + chat served by an OpenAI-compatible vLLM backend running a 27B Qwen model. Retain/consolidation/reflect all point at that one backend.

---

Summary

For one specific document, the asynchronous retain path (POST /v1/{tenant}/banks/{bank}/memories with async: true) hangs in retain_extract_facts indefinitely, while the synchronous POST /v1/{tenant}/banks/{bank}/memories/dry-run-extract extracts the identical text successfully in 10–30 seconds, repeatedly.

The two paths appear to build different prompts for the same input. I could not find a way to make the async path succeed for this document, and I eliminated the obvious explanations one at a time (below).

A second, distinct failure with the same symptom but a different cause is included as a footnote — for that one, switching retain_extraction_mode to chunks is a reliable workaround. For the primary bug, it is not.

---

Evidence

The sync path always succeeds

POST /memories/dry-run-extract, same content each time:

Attempt	Mode	Result
1	default (concise)	OK — 26.5s, 3 facts
2	chunks	OK — 30.3s, 3 facts
3	verbatim	OK — 10.3s, 1 fact
4	default, content only	OK — 28.0s, 3 facts
5	default, content + context	OK — 27.0s, 3 facts

The async path always hangs

POST /memories with async: true, same content, same bank:

Attempt	Config	Client bound	Outcome
1	default (concise)	300s	timed out, op still pending
2	bank retain_extraction_mode: chunks	420s	timed out, op still pending
3	default, fresh operation_id	600s	timed out, op still pending
4	default, different document_id	120s	timed out, no document created

Server-side, the operation sits in the same stage for as long as it is observed, and the built-in watchdog flags it:

[WORKER_TASK] [STUCK?] op=<id> type=batch_retain op_type=retain
              bank=<bank> age=776s stage=llm.openai.retain_extract_facts.attempt=1/5 stage_age=776s

[STUCK_STACK]  op=<id> type=batch_retain bank=<bank>
               age=624s threshold=600s stage=llm.openai.retain_extract_facts.attempt=1/5

Note attempt=1/5 never advances — the retry ladder does not engage while the call is outstanding.

What was ruled out

• The content itself — dry-run-extract parses it fine, 4/4, across all three extraction modes.
• The context string — controlled directly: content alone 28.0s/3 facts vs content + context 27.0s/3 facts. No difference.
• Queue congestion / a starved worker lane — attempt 3 ran after the lane had drained and other documents were completing normally; it still hung. Unrelated batch jobs on the same backend had finished.
• Document-level locking or serialization — attempt 4 used a brand-new document_id with a fresh operation_id. It hung identically and created no document, so the hang is not inherited from the earlier stuck operations bound to the original id.
• A dead backend — the vLLM container stayed healthy throughout and served other extractions (11 sibling documents in the same batch landed normally in 5–30s each).

The one time it landed, output was degraded

After several stuck operations eventually drained, the document appeared with 1 extracted fact, where dry-run-extract reliably produced 3 from the same text. That is consistent with the async path feeding the model a materially different (larger) prompt rather than simply being slower.

---

Hypothesis (not verified)

Labelled explicitly as a hypothesis — I have not read the source.

The async extractor may fold existing bank facts into the extraction prompt for deduplication or entity-linking, while dry-run-extract does not. At the time of these attempts the bank already contained facts extracted from a closely related document — same referenced snapshot name, same task id, overlapping entities. A dedup-augmented prompt built from those near-duplicate facts could plausibly push the model into a degenerate generation loop, which would explain all four observations at once: sync succeeds, async hangs, the hang is independent of document_id, and the eventual output is impoverished.

If that is the mechanism, the fix is probably a cap or a circuit-breaker on how much prior-fact context is injected, plus a generation-length guard.

---

Aggravating factor: a hung call holds a worker slot for 30 minutes

HINDSIGHT_API_RETAIN_LLM_TIMEOUT defaults to 1800 in this deployment. Because attempt=1/5 does not advance while the call is outstanding, a single hung extraction occupies its worker slot for the full 30 minutes before the retry ladder engages — and then may retry into the same loop up to four more times. In practice one pathological document degraded throughput for the whole instance for a long stretch.

Two smaller things noticed while diagnosing this, both of which made the failure harder to see:

• A completed operation is not proof that facts landed. In one case the parent operation reported unit_ids_count: 0 while the document had memory_unit_count: 1.
• GET /operations caps limit at 100 and returns HTTP 422 above that, whereas GET /documents accepts limit=200. Minor inconsistency, easy to trip over when enumerating stuck work.

---

Reproduction data

All content below is synthetic test data written for a demonstrator — no real people, systems, or business facts.

Primary case (async hangs, sync succeeds)

document_id: ana-p2-001, retained with context: "SPRO AI Brain synthetic demo seed" and string-valued metadata.

On 2026-08-23 agent:ana-metrics read the published Strategy snapshot snapshot-strategy-run-1 in the SPRO brain and found the open handoff candidate task:validate-metric-defs, which asks for the six candidate success-metric definitions v0.1 from S1 transformation-operating-model to be validated against real usage telemetry. Because A2 adoption-telemetry already holds that data, agent:ana-metrics picked the task up without a brief from Strategy and started the validation.

Tags on the item:

team:analytics, project:outcome-measurement, task:validate-metric-defs,
type:agent-report, status:active, source:agent-report,
visibility:spro, sensitivity:synthetic, agent:ana-metrics

To reproduce the relevant precondition, the bank should already contain a retained document referencing the same snapshot name and task id — in my case a completion report for the same task:validate-metric-defs, retained minutes earlier.

Footnote — a second, distinct hang where chunks mode is the workaround

This one behaves differently and is probably a separate bug; including it in case they share a root cause.

document_id: ana-p1-014.

Status change on 2026-08-14: epic JIRA-A1-100 for A1 outcome-measurement moved from active to blocked, reason given as undefined transformation-success metrics. Maya Chen (person:maya-chen) made the move and left the wireframe subtasks open rather than closing them.

Behaviour: hangs deterministically under the default concise extraction prompt — three independent attempts, including a dry-run-extract that timed out at 150s, while 58 sibling documents on the same backend extracted in 5–30s each. Unlike the primary case, every variant succeeds:

Variant	Result
verbatim mode	OK — 13.1s, 1 fact
chunks mode	OK — 35.9s, 2 correct facts
remove the inline (person:maya-chen) parenthetical	OK — 29.8s, 2 correct facts
first sentence alone	OK — 35.3s, 1 fact
second sentence alone	OK — 29.1s, 1 fact

Only the exact full text under concise loops. Workaround: set the bank's retain_extraction_mode to chunks, retain, restore. That preserves the source text verbatim and produces correct facts.

The distinction that matters: for ana-p1-014 the sync and async paths agree (both hang under concise), so it looks like a prompt/content interaction. For ana-p2-001 they disagree, which is why I think the async path is doing something extra.

---

What would help

1. Confirmation of whether the async extraction prompt differs from dry-run-extract — specifically whether prior bank facts are injected.
2. A generation-length or wall-clock guard on retain_extract_facts that fails the attempt rather than holding the slot until RETAIN_LLM_TIMEOUT.
3. attempt=n/5 advancing on a stuck call, so the retry ladder is reachable.
4. Optionally, surfacing "extraction produced 0 facts" as a distinct operation outcome rather than completed.