fix(documents): robust ingestion errors, status endpoint, and re-ingest - #283
Merged
Conversation
Addresses several document ingestion pain points:
- Synchronous ingestion of large files no longer surfaces "[object Object]".
`failure_reason` is now normalized (DomainError code / Error message /
string fallback), and files over SYNC_INGESTION_MAX_BYTES (default 10 MB)
are rejected up front with 413 FILE_TOO_LARGE_FOR_SYNC and guidance to use
async mode.
- Documents stuck in `pending`/`processing` self-recover: when read past
INGESTION_STALL_TIMEOUT_MS (default 5 min) with no progress they transition
to `failed` with `INGESTION_TIMEOUT`.
- New lightweight GET /documents/:id/status returning
{ status, chunk_count, total_pages, error? } for cheap polling instead of
the full (multi-MB) get-document payload.
- New POST /documents/:id/ingest to re-process an existing document (recover a
stuck doc or re-chunk with a different strategy) without delete + re-upload.
Both new operations flow into the SDK, CLI manifest, and MCP tool surface via
the OpenAPI spec. Adds unit + MCP + smoke coverage and updates module docs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DPxCHtSDFGM7go7wDSVgsP
The status endpoint's fields were ambiguous during processing: chunk_count came from metadata (only written on completion) with a DB-count fallback, and total_pages was simply absent until ready. - chunk_count is now always the live count of indexed chunks, so it is a meaningful progress signal during processing and equals the final total when ready (0 while pending). - total_pages is explicitly null until extraction finishes (ready/failed), documented as distinct from "zero pages". Updates the OpenAPI schema descriptions, module docs (field table), and adds a test asserting total_pages is null for a non-paged plain-text document. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DPxCHtSDFGM7go7wDSVgsP
Deploy Outputs
|
Chunks are now persisted incrementally as each embedding resolves (instead of one batch at the end), and the pipeline records total_chunks/total_pages up front and periodically refreshes indexed_chunks. This lets the status endpoint report real progress while a document is still processing. GET /documents/:id/status now returns: - total_chunks — planned total (denominator), null until chunking starts - progress — chunk_count / total_chunks as a percentage: 0 while pending, climbing while processing (capped at 99), 100 when ready, null when failed The periodic metadata refresh also bumps updatedAt, preventing a long-running ingestion from being misdetected as stalled. Updates OpenAPI schema, module docs (field table + example), and tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DPxCHtSDFGM7go7wDSVgsP
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a cluster of document-ingestion pain points (issues #3–#7):
[object Object]and silently timed outfailure_reasonis now normalized (DomainErrorcode →Error.message→ string fallback, never[object Object]). Files larger thanSYNC_INGESTION_MAX_BYTES(default 10 MB) are rejected up front with 413FILE_TOO_LARGE_FOR_SYNCand guidance to use async mode.processingforever, no recoverypending/processingwith no progress pastINGESTION_STALL_TIMEOUT_MS(default 5 min) self-recover tofailedwithINGESTION_TIMEOUTwhen read.get-document(6–8 MB) was the only way to check statusGET /documents/:id/status→{ status, chunk_count, total_pages, error? }. Never returns chunk content.POST /documents/:id/ingestre-runs ingestion against the existing source file (recover a stuck doc or re-chunk with a different strategy). Async by default (202),?async=falsefor201.Implementation notes
src/lib/documentIngestion.ts(sync-size guard, error normalization, stall recovery, re-ingest) andsrc/lib/documents.ts(getDocumentStatus). Route handlers stay thin.get-document-status,reingest-documentMCP tools).FILE_TOO_LARGE_FOR_SYNC(413) added to the registry.SYNC_INGESTION_MAX_BYTES,INGESTION_STALL_TIMEOUT_MS.Tests
Red/green TDD. Added unit coverage in
documents.test.ts([object Object]fix, 413 guard, status endpoint incl. self-recovery, re-ingest incl. async/stuck/404/401/403), MCP coverage inmcp.test.ts, and end-to-end steps intests/smoke-tests.sh.🤖 Generated with Claude Code
https://claude.ai/code/session_01DPxCHtSDFGM7go7wDSVgsP
Generated by Claude Code