Bug: accept_source_items embeds one document at a time, so a 500-item connector pass takes ~15 min and trips the host AcceptSourceItems deadline

[YellowSnnowmann]

Version or commit

tinymemory v1.14.1 (52836fd), running as the prebuilt module in openhuman main f5c5ba892.

Rust toolchain

rustc 1.96.1 (aarch64-apple-darwin)

Reproduction

1. In openhuman, connect a Gmail account with ≥500 messages. The host hands one pass of SYNC_PASS_MAX_ITEMS = 500 items to the connector, which delivers them in a single AcceptSourceItems call.
2. Watch memory_docs rows for source_gmail_<connection> appear at ~40 per minute (about 1.7 s per item), and mem_tree_chunks at the same pace.
3. Exactly 15 minutes after the connector logged run finished … stage="completed" pages=5 ingested=500, the host logs:

WRN [composio:bus] initial sync failed toolkit=gmail connection_id=… error=ingesting gmail records failed:
    call to `AcceptSourceItems` timed out after 900000ms

All 600 documents (500 + a second 100-item run) had landed by then; the "failure" is the caller giving up on a call that was still succeeding.

Expected behavior

A 500-item pass finishes well inside the host's 15-minute slow-call deadline, or at least is not bound by one embedding round-trip per document. Ingest throughput should scale with provider batch size, not with item count.

Actual behavior

crates/tinymemory-tinycortex/src/engine/mod.rs:2199 accept_source_items loops the items sequentially. Each iteration calls put_doc → upsert_document_presanitized (crates/tinymemory-core/src/store/namespace_store/documents.rs), which chunks the document (225 tokens) and embeds all of its vector_chunks in one provider call per document before the row and chunks are written, then ingest_connector_item_tolerated writes the tree chunk. Measured cost ≈ 1.7 s per item against the managed cloud embedder, so 500 items ≈ 850 s, right at the 900 s deadline.

Additional context

Options, roughly in order of preference:

1. Batch embeds across documents. Collect the chunk texts for the whole accept_source_items batch (or a bounded window of documents), embed them in provider-sized batches (the re-embed backfill already sizes to 1000 texts per request), then write rows and chunks. The per-key document_write_lock and the row/chunk ordering concern in documents.rs still hold per document.
2. Take the ingest-time embed off the request path. put_doc_light for connector items plus queue-driven embedding. Cheaper, but namespace vector search for those items is unavailable until the queue catches up, and today the queue's only embed path is itself starved (tinyhumansai/tinycortex issue linked from the parent).
3. Report progress instead of silence. Stream partial acknowledgements so the host's slow-call deadline can measure silence rather than total duration.

Host-side mitigation shipping first in openhuman (SYNC_PASS_MAX_ITEMS 500 → 200) keeps a pass inside the deadline but does not change the per-item cost. Related: #84 (embedder dimension/empty-vector contract). Parent issue with the full RCA and the Sources-row symptom: tinyhumansai/openhuman#6025.