feat(reparse): reuse chunks and processing artifacts

[0Tty0]

Summary

• reuse unchanged indexed chunk vectors during knowledge reparse with stable, knowledge-scoped identities
• provide a tenant-scoped durable processing artifact store with atomic put-if-absent semantics
• reuse canonical VLM OCR, caption, document embedding, DocReader, summary, generated-question, Wiki map, and GraphRAG extraction artifacts
• preserve current-run ownership when cached artifacts are rebound to chunks, knowledge items, and graph namespaces
• add corruption repair, payload limits, lifecycle counters, trace status, and scheduled retention for processing artifacts
• keep dynamic reduce/publication work live where cross-document state must not be cached

Why

Knowledge reparse previously removed existing chunks and indexes before rebuilding them, repeating expensive provider work and creating a destructive failure window. The new reconciliation flow preserves reusable results when normalized inputs and effective provider identities still match, while correctly invalidating changed content, model configuration, prompts, and output contracts.

The shared artifact layer freezes only successful ownership-free outputs. Every cache hit is validated and rebound to the current run, so reuse does not leak tenant, document, chunk, or graph ownership. Retry ordering and atomic winner semantics keep concurrent workers convergent without publishing partial state.

Implementation

Incremental chunk and embedding reuse

• derive stable chunk identities from knowledge scope and normalized content
• preserve indexed chunks only when content and effective embedding configuration match
• batch artifact reads and writes to avoid per-chunk database round trips
• deduplicate repeated text within a batch and restore the original result order
• validate cached and provider vectors for count, dimensions, and finite values
• reindex desired chunks before deleting stale vectors, images, graph ownership, and rows
• safely replace existing SQLite vectors when a stable source ID is reused

Processing artifact foundation

• key artifacts by tenant, stage, key version, and SHA-256 input fingerprint
• store small payloads inline and spill larger values to object storage through verified manifests
• enforce tenant-scoped access, configured payload limits, hash validation, corruption eviction, and atomic put-if-absent winners
• support PostgreSQL, MySQL, SQLite/Lite, and ParadeDB startup paths
• preserve the existing single-value path as the compatibility and large-object fallback

VLM and DocReader reuse

• key OCR and caption outputs by image bytes, effective non-secret model identity, prompt semantics, result kind, and canonicalizer version
• cache stable non-URL DocReader results by file bytes, effective reader identity, normalized metadata, parser overrides, and codec versions
• serialize ownership-free parse results with versioned streaming JSON and gzip
• bypass unstable URL, temporary, audio, ambiguous-reader, and unknown-override inputs
• rebind fresh child chunks and current-run storage metadata on every hit
• expose cache status through existing tracing metadata without logging content or cache keys

Chat and Wiki reuse

• key summaries and generated questions by exact rendered inputs, complete chat options, language, instructions, and non-secret model revision
• generate stable summary chunk IDs and deterministic occurrence-aware question IDs
• preserve unrelated metadata and publish desired question vectors before removing stale legacy IDs
• cache complete per-document Wiki map output while keeping reduce, taxonomy, publication, and finalization live
• validate entity schema, citations, source ordinals, slugs, dedup dependencies, UTF-8, and deterministic layout at the codec boundary

GraphRAG extraction reuse

• key extraction artifacts by tenant, chunk content, exact rendered messages and options, prompt/canonicalizer versions, and effective non-secret model revision
• store canonical ownership-free nodes and relationships, then bind the current chunk ID on decode
• replace one chunk's graph contribution atomically instead of appending duplicate ownership
• track node and relationship ownership by chunk so stale chunks can be retracted precisely
• fence graph writes by processing attempt to prevent superseded tasks from publishing
• migrate legacy namespace data before ownership-aware replacement
• recover interrupted namespace deletion before incremental cleanup
• serialize full namespace deletion with tokens and dirty-state recovery
• retain shared graph entities until their final contributing chunk is removed
• expose hit, miss, bypass, and provider-call status through the existing extraction span

Lifecycle operations and retention

• invalidate malformed cached values with observed-content compare semantics so concurrent repairs are not deleted
• repair corrupt concurrent winners while preserving the current provider result as a safe fallback
• encode authoritative object references and verify backend ownership before reads or deletion
• resolve exact historical storage backends for internal artifact reads and cleanup after disablement or soft deletion
• expose per-stage hit, miss, write, eviction, and error counters plus cache-status trace fields
• purge expired manifests and owned objects in bounded ID-ordered batches
• make retention interval, retention days, batch size, and maximum payload configurable with validated defaults and environment overrides
• retain failed cleanup entries for a later sweep and classify operational failures without exposing cached content or keys

Scope

This PR implements:

1. stable-ID chunk and embedding reuse within one knowledge item
2. tenant-scoped durable processing artifacts
3. VLM OCR and caption reuse
4. cross-document embedding reuse within the same tenant
5. DocReader parse artifact reuse for stable non-URL files
6. summary and generated-question reuse with deterministic current-run bindings
7. complete per-document Wiki map reuse with live dynamic-state validation and reduce
8. GraphRAG extraction reuse with ownership-aware replacement, attempt fencing, and precise stale cleanup
9. processing artifact validation, repair, observability, and scheduled retention

Administrative cache inspection/purge APIs and external dashboards remain follow-up work for #1679.

Validation

• go test -tags sqlite_fts5 -count=1 ./internal/types ./internal/application/repository ./internal/application/repository/retriever/postgres ./internal/application/repository/retriever/sqlite ./internal/application/repository/retriever/tencentvectordb ./internal/application/service/file ./internal/application/service/retriever ./internal/application/service ./internal/container
• CGO_LDFLAGS=-lssp go test -count=1 ./internal/config ./internal/types ./internal/application/repository ./internal/application/service/file ./internal/application/service
• focused repeated and race tests for processing artifacts, storage backends, VLM, embeddings, DocReader, chat enrichment, Wiki map, and graph extraction reuse
• go vet ./internal/config ./internal/types ./internal/application/repository ./internal/application/service/file ./internal/application/service
• gofmt
• git diff --check

Related to #1679.