release(02-2026): stable, functional, local RAG#38
Merged
Conversation
…val, hardening) (#31) * feat(ops): add readiness/index diagnostics and richer rag-status * test(ops): cover readiness drift/corruption and cli status * docs: note strict /api/ready drift detection in dense/hybrid * updated .gitignore - pr descs into /pr * feat(sql): add document identity columns + best-effort sqlite migration * test(sqlite): cover identity column migration, unique external_id, sha256 fill * docs: document document identity fields (external_id, metadata, hash, timestamps) * feat(docs): idempotent upsert by external_id (API/CLI) * test(docs): cover upsert idempotency (api/sql/cli) * docs: document docs upsert endpoint and rag-upsert-docs * docs: update roadmap with stabilization/refactor PR and incorporate quick-wins * refactor(structure): move api schemas/diagnostics and core helpers; keep shims * refactor(imports): prefer new module locations (app.schemas, core.services, app.diagnostics) * refactor(structure): drop root shims and enforce app/core module boundaries * docs: update roadmap and architecture after removing root shims * feat(ingestion): add txt/md/csv loaders with discovery + sniffing - Add TextFileLoader and MarkdownLoader - Add discover_files() limits (max files / bytes) for safe dir ingestion - Add loader factory with best-effort format detection (extension + bytes heuristics, optional python-magic) - Extend CSVLoader with delimiter sniffing and row_index metadata * feat(cli): add rag-ingest (dir/file) with SQL+FAISS consistency - Add CLI/entrypoint (files/dirs, limits, dry-run) - Ingest via external_id upsert per chunk and delete stale chunks by source prefix - Dense/hybrid: incremental vector updates, fallback rebuild on failure - Add SqlDocumentStorage.list_ids_by_external_id_prefix() helper - Add optional extra for python-magic * test(ingestion): cover rag-ingest + loader detection/discovery - CLI ingest: mixed dir, idempotency, and stale-chunk deletion - Factory detection: prefer content heuristics over misleading extensions - Discovery: enforce limits and handle broken symlinks - Update CSVLoader unit tests for row_index metadata * docs: document rag-ingest and optional python-magic - README: add rag-ingest to CLI commands and mention magic extra - custom usage guide: add a CLI ingestion section * fix(ingest): avoid external_id prefix collisions when syncing stale chunks Pass a delimiter-suffixed prefix when listing existing external_ids so /tmp/foo does not match /tmp/foo2. * feat(ingestion): configurable cleaning + deterministic chunking (chars_v1) - Add chunk_chars_v1 with stable boundaries (start/end offsets) - Make preprocess_text configurable via keyword options - Add settings flags for ingestion cleaning and chunk strategy - Provide helpers to build preprocess/chunk functions from Settings * feat(ingest): store chunk metadata (chunk_index/offsets/parent_doc_id) - rag-ingest now records chunk offsets and parent_doc_id in SQLite metadata - bootstrap uses Settings-driven cleaning/chunking for determinism * test(chunking): validate boundaries/overlap and ingestion metadata - Add unit tests for chunk_chars_v1 determinism, boundaries and overlap - Extend rag-ingest CLI test to assert prepared metadata (chunk_index, offsets, parent_doc_id) - Add coverage for preprocess_text options * feat(dedup): add chunk-level hash column + index and helper - Add Settings.ingest_chunker_version to version the dedup key - Add chunk_dedup_sha256 helper (sha256(cleaned_text + chunker_version + embedding_model_name)) - Extend documents schema with chunk_dedup_sha256 and a partial unique index - Allow upsert to persist chunk_dedup_sha256 when provided * feat(api): dedup /api/docs ingestion via chunk hash upsert - Clean + chunk texts using Settings-driven pipeline - Compute external_id as chunk hash (includes chunker_version + embedding model) - Upsert into SQLite and update FAISS incrementally in dense/hybrid mode * test(dedup): cover hash-based /api/docs idempotency and SQL constraint - Verify /api/docs dedup is idempotent across re-ingests - Verify changing ingest_chunker_version produces new inserts - Assert chunk_dedup_sha256 unique index is enforced * docs: document chunker version and dedup column - Add chunking strategy/version env vars to README and .env.example - Mention chunk_dedup_sha256 schema column * fix(db): run SQLite compatibility migrations for CLI/scripts CLI/scripts can be executed without starting FastAPI, so ensure the same best-effort SQLite migrations (AUTOINCREMENT + identity columns) run before using ORM-mapped queries. * feat(delete): add tombstones and delete-by-external-id in SQL repo - Add DocumentTombstone table (external_id unique) - Add SqlDocumentStorage delete_by_external_ids + tombstone helpers - Tombstones prevent deleted identities from reappearing on future ingestions * feat(api): delete by external_id with tombstones - Add /api/docs/delete_by_external_id endpoint - Filter tombstoned external_ids from /api/docs ingestion so deletes don't reappear - Reject tombstoned external_ids on /api/docs/upsert * feat(cli): rag-delete-external-ids - Add CLI command delete-external-ids (SQL + FAISS when dense/hybrid) - Respect tombstones during rag-ingest and block upserts for tombstoned external_ids - Add project script entrypoint rag-delete-external-ids * test(delete): cover external_id delete (tombstones) + ask_eval + rebuild - Sparse: delete external_id prevents re-ingest and removes results from ask_eval - Dense: delete external_id updates vector store and survives rebuild-index * docs: document delete-by-external-id (API/CLI) - Add rag-delete-external-ids and /api/docs/delete_by_external_id to usage guides * feat(index-manifest): persist manifest and report drift in ready/status * chore(mypy): avoid optional python-magic typing issues * test(index-manifest): cover manifest drift and rebuild recovery * docs(index-manifest): document index_manifest and drift detection * test(index-manifest): unit coverage for manifest helpers * feat(reranker): optional overlap reranker behind settings * feat(observability): structured logs + domain metrics for ingest/query * feat(eval): add rag-eval CLI with versioned dataset and regression gate * test(eval,reranker): cover offline eval gate and reranking behavior * docs(pr10): document rag-eval, monitoring, and optional reranker * PR11: Hardening de consistencia transaccional y concurrencia (SQL + índice) (#27) * fix(consistency): preflight manifest drift before index mutations * fix(maintenance): raise explicit multi-store inconsistency error on double failure * ci: add windows smoke tests for faiss persistence and consistency paths * fix: prevent SQL/vector drift when embedding fails Precompute dense/hybrid embeddings before SQL upsert in API and CLI flows so provider failures cannot leave partially committed SQL without vector updates. Add SqlDocumentStorage.get_existing_doc_states_by_external_id to embed only inserts/content updates (preserving idempotent no-op behavior). Add regression tests for /api/docs, /api/docs/upsert, rag-upsert-docs, and rag-ingest to verify no SQL persistence on embedding failures. * fix: harden rag service reload token writes Use unique temp files + fsync when writing .rag_service_reload_token to avoid .tmp path collisions across workers. Add regression test ensuring _write_reload_token does not reuse temp source paths across consecutive writes. * docs(pr): add PR11 hardening consistency description * fix(api): serialize multi-store writes and harden generation params Prevent concurrent mutating API operations from interleaving SQL and FAISS updates by running write paths under a shared cross-process lock. Add strict bounds for generation sampling params (temperature/top_p/max_tokens) in OpenRouter requests and AskEval top_p validation. Add regression tests for concurrent upsert consistency and invalid sampling parameter rejection. * chore: fix import ordering and include roadmap updates * fix(api,lock): prevent duplicate ingest pass and fail closed on lock acquisition * fix(cli): serialize mutating commands with shared multi-store write lock * docs(pr11): include integral audit findings, fixes, and verification * fix(faiss): fail closed on missing manifest and lock acquisition * fix(api,cli): always invalidate cached rag service after mutating attempts * chore(lint): align isort with ruff and normalize import ordering * docs(pr11): document new critical audit findings and validations * fix(delete): deduplicate external_ids to prevent tombstone integrity failures * fix(openai): enforce request timeouts with compatibility fallback * chore(format): normalize TYPE_CHECKING import block in evaluation service * docs(pr11): add critical audit findings for delete dedup and OpenAI timeouts * fix(security): enforce runtime API-key guard for non-local requests * fix(coordination): align lock and reload token paths across processes * docs(pr11): record critical findings 14-15 and runtime security notes * refactor: centralize dense upsert consistency flow and shared locking/client helpers * test: add regression coverage for shared dense sync, manifest config, and OpenAI client fallback * docs: update PR11 audit notes and include curso indices * fix(security): fail closed when client host is unavailable * fix(consistency): make dense delete paths lazy-load embedders * fix(ingest): honor no-follow-symlinks for symlink directory inputs * refactor(cli): centralize dense embedder construction across commands * docs: clarify symlink ingest scope and dense delete behavior * polish * refactor(core): simplify lock orchestration and dense precompute flow * test(app): cover blocking worker env parsing and execution path * perf(ingest): avoid duplicate format detection and reuse precomputed result * perf(text): precompile whitespace regex in preprocessing * perf(ingest): batch rag-ingest mutations to reduce upsert/lock churn * perf(sparse): cache docs in memory and remove per-query SQL get * chore(lint): move NDArray import under TYPE_CHECKING * perf(sparse): avoid duplicate SQL loads when building retrievers * fix(security): enforce API key for forwarded non-local requests * fix(consistency): harden multi-store script locking and fail-fast build-index * fix(security): enforce API-key guard for RFC7239 Forwarded hosts * fix(consistency): preflight vector mutability before SQL deletes * fix(sqlite): tolerate duplicate-column races in identity migration * ci: harden workflow gates and pin actions by SHA * build: make Docker reproducible and enforce strict security target * docs: document CI gates and contributor verification workflow * refactor: centralize composition, harden multistore deletes, and split eval layering * fix(ingest): handle unicode text detection and unreadable files Treat non-ASCII UTF-8 text as textual content during loader detection by using Unicode-aware printability checks. This prevents valid multilingual .txt files from being misclassified as unknown and silently skipped by rag-ingest. Also harden detect_file_format to fail soft on unreadable files (e.g. permission errors), returning unknown/read-error instead of raising and aborting the full ingestion run. Add regression tests for: - UTF-8 non-ASCII plain text detection - read-head permission error handling - end-to-end CLI ingest with non-ASCII text Update README ingestion notes to document the behavior. * refactor(composition): unify runtime wiring policy across api, cli, and scripts Centralize adapter-selection policy in app.composition and reuse it from API factory, ask_eval retriever wiring, CLI dense embedder paths, and bootstrap/build-index scripts. Changes: - Add DEFAULT_DENSE_BACKEND_MESSAGE in composition. - Make build_dense_embedder_from_settings use the shared default when message is omitted. - Add build_retriever_with_default_embedder_from_settings to remove duplicated dense-embedder closure wiring. - Add resolve_preferred_llm_provider and reuse it in app.factory. - Update API/CLI/scripts to call shared composition policy helpers. - Add unit tests for new composition helpers and provider policy. This keeps existing monkeypatch seams in api/factory tests while consolidating policy decisions in one reusable module. * refactor(app): extract docs and index use-cases from api router Move business orchestration for /api/docs*, /api/docs/upsert, /api/docs/delete*, and /api/index/rebuild into app-layer services. Changes: - Add app/services/docs.py with sync use-cases for ingest/upsert/delete flows and typed summaries. - Add app/services/index.py with rebuild_index_sync orchestration. - Keep api_router transport-focused: request normalization, HTTP mapping, observability, and response serialization only. - Preserve existing monkeypatch seams by injecting adapter factories/callbacks from api_router into services. - Update architecture docs to reflect app/services docs/index layering. Validation: - ruff check . - mypy . - pytest -q (335 passed, coverage >= 85%). * refactor(cli): partition commands by bounded context modules - split monolithic cli.py into domain command modules: docs, index, eval, server\n- keep cli.py as composition root + entrypoint wrappers + shared hooks\n- preserve command surface/flags and existing monkeypatch seams\n- add registry/entrypoint tests for command wiring\n- update architecture docs with CLI layering * refactor(factory): replace reload token file with DB-backed system_state version - introduce SystemStateStorage adapter in SQLAlchemy persistence\n- persist rag service cache version in system_state (key: rag_service)\n- remove filesystem token invalidation from app factory\n- keep process-local cache maxsize=1 while enabling cross-process invalidation via DB version\n- add focused tests for factory invalidation and system_state storage behavior\n- update architecture docs to reflect DB-backed invalidation * refactor(blocking): add task-type pools with explicit pending limits - partition run_blocking work into task types: default, mutation, network, eval\n- add per-type worker and queue limits via env-configurable policies\n- enforce max pending per task type to prevent unbounded contention\n- route API run_blocking calls with explicit task_type labels\n- extend blocking and multistore tests for routing, queue-full behavior, and slot release guarantees\n- document async/sync boundary policy in architecture guide * test(app): add mandatory chaos scenarios for mutating operations - add chaos coverage for vector write failure during upsert\n- add lock acquisition failure scenario for mutating upsert path\n- reproduce crash window: SQL committed while vector upsert+rebuild fail\n- add rebuild-index failure scenario with cache invalidation assertion * refactor(app): introduce docs/index application ports to reduce infra coupling - add app-level ports module with DocsMutationPorts and IndexMutationPorts contracts\n- refactor docs and index use-cases to depend on port bundles instead of infra callables\n- centralize docs/index dependency wiring in api_router via _docs_mutation_ports/_index_mutation_ports\n- keep existing monkeypatch seams intact by resolving router symbols at runtime\n- add wiring tests for docs/index ports binding\n- update architecture docs to document new app ports layer * fix(security): fail closed on ambiguous proxy forwarding chains * fix(api): return 502 for malformed openrouter responses * fix(cli): validate duplicate external_id before embedding work * refactor(app): split transport routers and enforce typed provider errors Introduce typed cross-layer LLM errors in core and remove FastAPI exceptions from infra adapters. Add app-layer HTTP error mapping, extract OpenRouter into dedicated router/service, and move docs/index mutation port wiring into reusable builders for API/CLI. Expand tests for error mapping, OpenRouter service behavior, mutation port builders, and updated adapter/router mappings. * refactor(cli): reuse app docs services and add multiprocess lock regression Route docs CLI mutations through app services and shared mutation port builders to remove duplicated orchestration logic. Add real spawn-based integration coverage for cross-process write-lock serialization and update architecture docs for bounded routers and error-layer boundaries. * refactor(api): reduce root router to composition and split bounded routers Extract health, rag, docs and index endpoints into dedicated router modules and move runtime wiring/composition helpers to app/wiring.py. Migrate ingest schemas into app/schemas and adjust unit tests to patch the new seams (wiring and bounded routers) while preserving endpoint behavior. Update architecture docs to reflect api_router composition-only role and new router/wiring structure. * hotfix-refactor: arch * chore(rescue): snapshot mixed worktree before pr04 split * update(ci): splitted branches globbing patterns on CI for granular control on trigerage. Included release** pattern * add(pre-commit): included pre-commit in TO PROJ. deps * config * ci(security): pin safety v2 and use stable check command * updated doc, repo structure * fix(working): fixing pre-commit + CI + bandic (sec) trigering and execution * fix(working): fixing pre-commit + CI + bandic (sec) trigering and execution * fix(working): fixing pre-commit + CI + bandic (sec) trigering and execution * fix(working): fixing pre-commit + CI + bandic (sec) trigering and execution
* refactor: replace assert guards with explicit RuntimeError + low-risk cleanups - Replace all production `assert x is not None` with explicit `if x is None: raise RuntimeError(...)` across sql_.py, index.py, composition.py, factory.py, docs_ingest.py — asserts are disabled under `python -O`; the new form is visible in all execution modes. - Extract `_to_domain_document()` module-level helper in sql_.py; eliminates duplicate 11-line ORM→domain mapping in `get()` and `get_all_documents()`. - Replace `getattr(db_doc, "field", None)` fallbacks with direct attribute access in sql_.py; all columns are explicitly mapped in Document ORM model — getattr implied optional columns that are guaranteed present. - Add `FaissIndex._locked_write()` context-manager abstracting the repeated pattern `with self._state_lock, _exclusive_file_lock(self._lock_path)` (5 call sites → 1 definition). - Add `Settings.ingest_batch_size` field (default 64, env-overridable via INGEST_BATCH_SIZE); removes magic number hardcoded in `_execute_ingest_batches`. All changes are behaviour-preserving. lint/mypy clean. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(sql): extract _DocumentChanges + _detect_document_changes, add change-variant tests Motivation: the upsert change detection block mixed four independent boolean checks inline — hard to test in isolation and easy to introduce regressions when adding a new field. Changes: - Add module-level `_DocumentChanges` frozen dataclass with `any_changed` property. - Add module-level `_detect_document_changes(db_doc, item, new_content, new_sha)` function, placed after `SqlDocumentStorage` so `SqlDocumentStorage.UpsertDoc` is already defined in scope. - Replace the 14-line inline detection block in `upsert_documents_by_external_id` with a single `changes = _detect_document_changes(...)` call. - Remove now-redundant local `content_changed` variable; reference `changes.content`. Tests added (test_sql_upsert_documents_by_external_id.py): - `test_upsert_metadata_only_change` — metadata diff → action=updated, content_changed=False - `test_upsert_source_only_change` — source diff → action=updated, content_changed=False - `test_upsert_dedup_only_change` — dedup diff → action=updated, content_changed=False These variants were not covered; they exercise all four branches of `_DocumentChanges`. lint/mypy clean, 6/6 upsert tests pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(ingest): replace tuple aliases IngestPlan + BatchSyncResult with frozen dataclasses Motivation: positional tuple unpacking of 4 and 7 fields respectively makes call sites opaque — a field reorder would silently produce wrong behaviour. Changes: - Add `from dataclasses import dataclass` import. - Replace `IngestPlan = tuple[Path, str, tuple[Any, ...], tuple[str, ...]]` with `@dataclass(frozen=True) class IngestPlan` with named fields (file_path, file_prefix, items, desired_external_ids). - Replace `BatchSyncResult = tuple[int, int, int, bool, int, int, list[...]]` with `@dataclass(frozen=True) class BatchSyncResult` with named fields (inserted, updated, unchanged, rebuilt, deleted_stale, ingested_chunks, stale_by_file). - Update `_build_file_ingest_plan`: return `IngestPlan(...)` instead of bare tuple. - Update `_build_ingest_plans`: `plan[2]` → `plan.items`. - Update `_collect_batch_items_and_stale`: replace 4-way destructuring `for a,b,c,d in` with `for plan in` + attribute access. - Update `_ingest_batch_sync`: return `BatchSyncResult(...)` instead of 7-tuple. - Update `_execute_ingest_batches`: replace 7-way destructuring with `batch = ...; batch.field`. Behaviour: identical. lint/mypy clean. 82 ingest-related unit tests pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(factory)+docs(settings): consolidate PR-A/PR-D and runtime defaults Includes all pending modified files in this branch, as requested. - PR-A: extract _collect_container_overrides() from _build_container() in app/factory.py while preserving monkeypatch seams, override keys, and resolution order. - PR-D: formalize INGEST_BATCH_SIZE in .env.example and README (config table + ingestion tuning section). - Add settings validator tests for ingest_batch_size bounds: accepts 1/512 and rejects 0/513. - Consolidate Ollama default model/docs updates to lfm2.5-thinking across settings, README, docker-compose comment, and custom usage guide. --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
- README.md: update FastAPI badge 0.111 → 0.124 - README.md: add cli_commands/ to project structure tree; fix scripts/ description (only sample_data_ingestion.py, not bootstrap) - docs/USAGE.md: sessionmaker(bind=engine) → sessionmaker(engine) (SQLAlchemy 2.x removed the bind= keyword from sessionmaker) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
architecture.md: document golden rule (delete app/ → core/services still work), current compliance (ingest/import use cases accept Sequence[str]/bytes), and the router's responsibility to convert HTTP types before use-case call. CHANGELOG.md: add [1.1.0] entry covering security CVE fixes, dep bumps, port contract changes, refactors, and tooling additions from the sprint. Clear stale [Unreleased] content (app/services/* references predated the app/contracts/ rename already shipped in 1.0.0). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tAPI BREAKING CHANGE: the `app/` package is removed entirely. - Use cases move to `core/use_cases/` (transport-agnostic). - DI composition root moves to `composition/` (transport-neutral). - HTTP adapter lives in `http/` (FastAPI optional via `[server]` extra). - Observability moves to `infrastructure/observability/`. - Blocking executor moves to `infrastructure/concurrency/`. - Storage profiles move to `core/domain/profiles.py`. - ASGI entry point: `local_rag_backend.http.main:app`. - `AskEvalConfigLike` Protocol replaces concrete Pydantic schema in container public signatures. - `map_runtime_error` moves from `http/error_mapping.py` to `core/use_cases/errors.py`. - Layer dependency rules enforced by architecture guard tests. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Architecture guard tests now scan `http/routers/` (not defunct `app/routers/`). - `test_http_routers_do_not_import_infrastructure_directly` refined to allow cross-cutting infrastructure (concurrency, observability) while still blocking direct imports of persistence, retrieval, LLM, and embedding adapters. - Stale path comment removed from `http/api_router.py`. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…g injectable __init__
…ootstrap contract
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Back-merge
release/02-2026intodevelopafter thev1.3.0release, including release stabilization, architecture refactors, durability fixes, docs alignment, and CI hardening.What’s Included
1.3.0inpyproject.toml.1.3.0(dated 2026-03-03).v1.3.0tag updated to include the final release fixes.--extra serversofastapi/uvicornimports are available in CI.release/02-2026(111 commits total).Notable Technical Changes
core/use_cases,composition, andhttpboundaries.Validation
ruff check src testsruff format --check src testsmypy srcpytest -q->499 passed, 1 skipped, coverage87.09%(>= 85%)release/02-2026after CI fix:22618332389->successBreaking/Behavior Notes
release/02-2026.Merge Intent
This PR keeps
developin sync with the shipped release line (v1.3.0) and avoids branch drift for upcoming work.