Chore/curation sync main 3 10 1

[danhdoan]

Summary

• Problem: sync main 3.10.1

Test plan

• Coverage added:
  • Unit test
  • Integration test

User-visible changes

None

Checklist

• Tests added or updated and passing (npm test)
• Lint passes (npm run lint)
• Type check passes (npm run typecheck)
• Build succeeds (npm run build)
• Commits follow Conventional Commits format
• Documentation updated (if applicable)
• No breaking changes (or clearly documented above)
• Branch is up to date with main

Risks and mitigations

None

[github-actions]

Claude finished @danhdoan's task in 8m 12s —— View job

---

PR Review: Chore/curation sync main 3.10.1

Tasks

• Gather context and understand the PR
• Review agent LLM and generator changes
• Review abstract generator batch processing additions
• Review abstract queue changes
• Review dream state/executor changes
• Review curate executor changes
• Review test files for quality and coverage
• Post inline comments for specific issues
• Post overall summary with score

---

Summary

This PR integrates five feature branches from main 3.10.1 into the curation-enhancement line. The dominant themes are prefix-cache poisoning prevention, batched abstract generation, and deferred stale-summary propagation. The code is well-structured, well-documented, and accompanied by solid test coverage. Below is a feature-by-feature breakdown.

---

1. Prefix Caching — ai-sdk-content-generator.ts, ai-sdk-message-converter.ts, agent-llm-service.ts

What changed:

• prependCachedSystemMessage moves the system prompt from the top-level system: parameter into the messages array as a role: 'system' entry with providerOptions.anthropic.cacheControl: ephemeral. This is the correct way to pin Anthropic's prefix cache to the system block when using the AI SDK, since the system parameter does not propagate providerOptions.
• toAiSdkTools now attaches cache control to the last tool definition, which causes Anthropic to cache the entire tool block and everything before it. Non-Anthropic providers silently ignore the anthropic namespace. ✅
• buildDateTimePrefix is extracted as a standalone utility; refreshDateTime is removed. Timestamps now go into the user message rather than the system prompt — preventing them from breaking the prefix-cache bookmark. Well-documented rationale.

Observation: buildDateTimePrefix takes now: Date = new Date() as a parameter, which makes it correctly testable and avoids the "datetime in system prompt" cache-poisoning concern highlighted in the existing codebase comments.

---

2. Batch Abstract Generation — abstract-generator.ts, abstract-queue.ts

What changed:

• generateFileAbstractsBatch batches up to 5 files into 2 parallel LLM calls (one L0, one L1) instead of 2N per-file calls.
• CDATA wrapping (wrapCdata) prevents file content that contains </document> or </file> markers from breaking the XML envelope — the CDATA-within-CDATA escape ]]]]><![CDATA[> correctly neutralises embedded ]]> sequences.
• parseBatchedTags anchors on <file path> openers rather than </file> closers, which prevents a literal </file> in model prose from prematurely terminating the slice and orphaning the inner tag. This is a subtle but important design choice.
• escapeXmlAttr properly escapes the path attribute value.
• AbstractGenerationQueue migrated from per-item to per-batch processing with BATCH_SIZE_CAP = 5.

Nitpick (abstract-queue.ts lines 213–220): Empty strings are written unconditionally when the model produces no content for a path:

await Promise.all(results.flatMap((r) => {
  const abstractPath = r.contextPath.replace(/\.md$/, '.abstract.md')
  const overviewPath = r.contextPath.replace(/\.md$/, '.overview.md')
  return [
    writeFile(abstractPath, r.abstractContent, 'utf8'),
    writeFile(overviewPath, r.overviewContent, 'utf8'),
  ]
}))

An empty .abstract.md is semantically different from a missing one — downstream search/manifest consumers may render a blank entry rather than skipping it. The existing fail-open semantics are preserved by simply filtering:

...(r.abstractContent ? [writeFile(abstractPath, r.abstractContent, 'utf8')] : []),
...(r.overviewContent ? [writeFile(overviewPath, r.overviewContent, 'utf8')] : []),

Fix this →

Dedup note (abstract-generator.ts line 250): The items.map() return at the end correctly maintains output alignment with the original input array (including duplicates), while the byPath Map ensures only the last (most recent) content for each duplicate path is sent to the model. This is the right tradeoff for queue callers that expect index-aligned results. ✅

---

3. Stale Summary Deferral — dream-state-schema.ts, dream-state-service.ts, dream-trigger.ts, dream-executor.ts, curate-executor.ts

What changed:

• DreamStateSchema adds staleSummaryPaths (array of {path, enqueuedAt} entries), totalDreams, and lastDreamLogId.
• DreamStateService gets drainStaleSummaryPaths (atomic RMW that clears the queue and returns the snapshot) and enqueueStaleSummaryPaths (dedup-aware, mutex-serialized append). The atomic drain is load-bearing: any enqueue after the drain returns sees an empty queue and appends independently — the earlier "snapshot + clear-later" approach lost same-path enqueues that slipped in during propagation. ✅
• DreamTrigger gate 2 now bypasses the minCurations guard when staleSummaryPaths.length > 0. Without this, low-activity projects (fewer than DEFAULT_MIN_CURATIONS = 3 curations in 12 h) would strand _index.md regeneration work indefinitely. ✅
• DreamExecutor step 5 merges the drained queue snapshot (source A) with the dream's own snapshot diff (source B) into an A∪B set before calling propagateStaleness. A path touched by both runs exactly once. The drain is atomic — concurrent curate enqueues during propagation append to the now-empty queue, not into the active set, ensuring they survive into the next cycle. ✅
• CurateExecutor.propagateAndRebuild enqueues changed paths to DreamStateService instead of inline LLM summary propagation, keeping curate's hot path free of LLM calls. The separate try/catch blocks for the dream-state enqueue and manifest rebuild ensure an error in one does not suppress the other. ✅

Design note (dream-executor.ts lines 187–209): drainedSnapshot is declared before the try block and populated inside it. If drainStaleSummaryPaths() itself throws, drainedSnapshot stays empty [] and the catch block's drainedSnapshot.length > 0 guard correctly skips re-enqueue — there was no successful drain to restore. Clean and correct. ✅

---

4. Review-Disable Propagation — dream-executor.ts

What changed:

• reviewDisabled is propagated from the daemon-stamped task options through DreamExecuteOptions → runOperations → createReviewEntries → formatResult. When true, reviewBackupStore is omitted from consolidate/prune so review-backup directories stay empty, and formatResult suppresses the "X operations flagged for review" output.
• reviewEntriesWritten flag guards against the race where step 7 (dreamStateService.update) throws after step 6b already wrote review entries. Without this guard, the catch block's createReviewEntries call would duplicate review items in brv review pending. ✅
• createReviewEntries correctly refactored from positional parameters to an object parameter (>3 params → object, per coding standards). ✅

---

5. Pre-pipeline Recon — curate-executor.ts

What changed:

• recon is computed synchronously before the agent loop starts and injected as a sandbox variable. The prompt explicitly instructs the agent not to call tools.curation.recon — saving a full LLM iteration for a deterministic helper. ✅
• The comment correctly notes that the metadata argument is currently unused by the recon helper but is passed for signature compatibility. Acceptable documentation.

---

Test Coverage

The test suite is thorough:

• abstract-generator-batch.test.ts: covers happy path, out-of-order responses, omitted files, malformed output, CDATA injection safety, CDATA terminator escaping, dedup, and call-count efficiency (exactly 2 LLM calls per batch). ✅
• abstract-queue.test.ts: covers batching, drain partial flush, retry backoff tracking in getStatus(), status file writes, drain-not-resolved-during-retry. ✅
• dream-state-service.test.ts: covers concurrent incrementCurationCount (10 concurrent, no lost updates), shared-instance serialization (two instances same file), step-7 reset race, enqueueStaleSummaryPaths dedup / timestamp preservation, drainStaleSummaryPaths atomicity. ✅
• dream-executor.test.ts: covers A∪B merge invariant, partial run review-entry preservation, reviewEntriesWritten double-write guard. ✅

---

Minor Issues

File	Location	Severity	Issue
abstract-queue.ts	L213–220	Nitpick	Empty strings written to .abstract.md/.overview.md when model produces no output — creates blank artifact files

---

Overall Score: 4 / 5

The design decisions are sound, well-justified in comments, and backed by comprehensive tests. The A∪B merge for stale summary paths and the atomic drain are the correct solutions to the concurrency problem. The main nit is the unconditional empty-string file write. Everything else is production-grade.

[ncnthien]

not really good but I think we need to approve now