fix(retain): keep oversized items in one async child to stop FK race (#1795)

[nicoloboschi]

Summary

Fixes #1795. submit_async_retain was splitting oversized retain payloads into N independent async_operations rows that all shared one document_id. Workers have no per-document gate for retain (the busy-bank guard in claim_tasks only covers consolidation), so siblings ran concurrently — each entered handle_document_tracking with is_first_batch=True and cascade-deleted the previous winner's memory_units. The loser's final ANN pass then inserted memory_links referencing now-deleted units, tripping fk_memory_links_from_unit_id_memory_units. Concurrent siblings also exhausted OS thread budgets via per-child sentence-transformer pools (libgomp resource-unavailable failures) and left partial document state visible to dry-run skip checks.

This is fix #2 from the issue's suggested fixes: do not pre-split a single document across independent child operations; let the in-process splitter handle intra-document chunking sequentially.

Approach

• New _split_contents_into_async_children helper for the async submit path. It packs items into children by token budget but never fragments a single item across children — oversized items go into their own one-item child holding the full un-chunked content.
• The worker's existing in-process splitter (retain_batch_async → _split_contents_into_sub_batches) re-chunks that content sequentially inside one worker slot with correct is_first_batch=(i==1) semantics — the same path that already enforces SELECT … FOR UPDATE + content-hash gating between batches of one call.
• Small items still pack together so genuinely independent inputs keep cross-worker parallelism.
• Metadata field names (num_sub_batches, sub_batch_index, total_sub_batches) and the parent/child operation structure are unchanged, so dashboards and status APIs are untouched.

Test plan

• 8 pure-Python tests for the new helper (tests/test_batch_chunking.py):
  • single oversized item → exactly 1 child holding the full content
  • metadata preserved through the helper
  • small items packed by budget
  • mixed small + oversized → small packed, oversized isolated
  • multiple oversized items → one each
  • single small item, empty input, oversized-at-boundaries
• 3 integration tests against the real DB (tests/test_async_batch_retain.py):
  • test_oversized_single_item_creates_one_child_not_many — submits one oversized doc and asserts the async_operations table has exactly one retain row whose task_payload.contents holds the un-chunked content. Empirically verified to fail on the pre-fix code with Expected 1 child for an oversized single item, got 7.
  • test_oversized_single_item_drains_without_fk_violation — drives a worker drain end-to-end and asserts no memory_links rows have orphan FKs in either direction — the exact invariant pre-fix code violated.
  • test_oversized_item_among_small_items_keeps_small_items_packed — confirms the parallelism optimization for genuinely-independent items isn't lost.
• Full tests/test_async_batch_retain.py + tests/test_batch_chunking.py suite passes (34 tests).
• Sanity sweep of tests/test_document_tracking.py, tests/test_op_cancellation.py, tests/test_async_retain_tags.py (22 tests) passes.
• ./scripts/hooks/lint.sh clean.