feat(core,backends): connectors framework — SPI evolution + filesystem/S3/GCS (Step 1.2)#55
Merged
officialCodeWork merged 2 commits intoMay 24, 2026
Conversation
…m/S3/GCS (Step 1.2) Evolves the Connector SPI to ship a resumable watermark with each discovered document, and lands three built-in connectors plus a shared internal crawler. Core (rag_core): - New ConnectorState frozen model (connector_id, tenant_id, cursor, last_seen_at, extra) added to rag_core.types and dist/schemas/. - Connector.list_documents now returns AsyncIterator[(Document, ConnectorState)] — replaces the bare `since: datetime` argument. Resumable mid-stream. - NoopConnector + contract tests updated with resume/watermark/tenant-mismatch coverage. spi_signature.py linter passes unchanged. Backends (rag_backends.connectors): - Internal Crawler[Raw] base class (iteration scaffold, fail/skip error policy, monotonic watermark bookkeeping). Not part of the SPI surface. - FilesystemConnector: cross-platform pathlib walk, sha256 hashing, absolute-path resume cursor, mime detection. - S3Connector: aioboto3 paginator, ListObjectsV2 continuation token, ETag-based content hash, MinIO-compatible. - GCSConnector: gcloud-aio-storage, GCS REST nextPageToken cursor; behind the [gcs] extra (gcloud-aio-storage>=9.3). Tests: - tests/contract/test_connector.py — 8 tests (resume, watermark monotonicity, tenant-mismatch). - tests/backends/test_filesystem_connector.py — 9 unit tests, pure-Python. - tests/integration/test_s3_connector.py — MinIO; skip-if-no-service. - tests/integration/test_gcs_connector.py — real GCS; skip when ADC absent. Docs: - docs/reference/connectors.md — public usage for all three connectors. - docs/architecture/connectors.md — SPI shape rationale, Crawler design, policy boundary (connectors are upstream of PolicyEngine; enforcement lives in the Step 1.10 ingest pipeline), CDC interaction, extension points. - docs/README.md updated. Tracker: - Step 1.2 flipped to 🚧 with branch + key deliverables. - Phase 1 counts: 8→9 done, 8→7 remaining. Total: 21→22 done, 63→62. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…tra) CI runs `uv sync` without the `[gcs]` extra, so `gcloud.aio.storage` is absent and mypy --strict fails on the GCSConnector import. Add `gcloud.*` to the missing-imports allowlist alongside the other optional/untyped backends (aioboto3, asyncpg, etc.). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2 tasks
officialCodeWork
added a commit
that referenced
this pull request
May 24, 2026
chore(tracker): sync Step 1.2 → ✅ and log PR #55
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ConnectorSPI to ship a resumableConnectorStatewatermark with each yielded document —list_documentsnow returnsAsyncIterator[(Document, ConnectorState)]. The oldsince: datetimeargument is gone.ConnectorStatetype (connector_id,tenant_id,cursor,last_seen_at,extra); JSON Schema regenerated.Crawler[Raw]base class inrag_backends.connectors._base— iteration scaffold,"fail"/"skip"error policy, monotonic-watermark bookkeeping. Not part of the public SPI surface.FilesystemConnector— cross-platformpathlibwalk, SHA-256 hashing, sorted absolute-path resume cursor.S3Connector—aioboto3paginator, ListObjectsV2 continuation-token cursor, MinIO-compatible (reuses existing dep).GCSConnector—gcloud-aio-storageasync client, GCS RESTnextPageTokencursor; behind the[gcs]extra.PolicyEngineboundary documented: connectors are upstream of policy; enforcement lives at the ingest call site (Step 1.10). Matches the pattern set by the storage backends.Tests
tests/contract/test_connector.py— 8 conformance tests covering empty/seeded listing, tenant isolation, fetch, resume-from-cursor, watermark monotonicity, tenant-mismatch rejection.tests/backends/test_filesystem_connector.py— 9 pure-Python unit tests, no services required.tests/integration/test_s3_connector.py— runs against MinIO via the dev stack; skipped automatically when MinIO is unreachable.tests/integration/test_gcs_connector.py— runs against real GCS; skipped whenGOOGLE_APPLICATION_CREDENTIALSorTEST_GCS_BUCKETare absent.spi_signature.pyandtests/policy/coverage.pylinters pass unchanged.uv run pytest tests/ packages/ -x -q→ green (integration suites skipped — no services running locally).uv run mypy packages/core packages/backends apps/gateway→ clean.uv run ruff check+format --check→ clean.Documentation
docs/reference/connectors.md— public usage: filesystem / S3 / GCS examples, resumable-crawl pattern, error table.docs/architecture/connectors.md— SPI shape rationale (why(doc, state)tuples beat a separatecheckpoint()),Crawlerdesign, policy boundary, CDC interaction (Step 1.9), extension points.docs/README.md— index entries added underarchitecture/andreference/.Test plan
uv run pytest tests/ packages/ -x -quv run mypy packages/core packages/backends apps/gatewayuv run ruff check . && uv run ruff format --check .uv run python -m rag_core.gen_schemas dist/schemas/produces only the newConnectorState.jsontask devthenuv run pytest tests/integration/test_s3_connector.py -m integration)ubuntu-22.04,macos-14,windows-latest🤖 Generated with Claude Code