|
| 1 | +# Cloud-Native KB Ingestion — Tenant Ingestion Model |
| 2 | + |
| 3 | +> **Status — MVP operational model.** This guide documents Sub E of Epic [#11720](https://github.com/neomjs/neo/issues/11720): how an external tenant gets its repository content into a cloud-deployed Knowledge Base without tacit Neo maintainer knowledge. The substrate it names was delivered by Epic [#11624](https://github.com/neomjs/neo/issues/11624); this guide defines the operator-facing model on top. |
| 4 | +
|
| 5 | +## Decision Summary |
| 6 | + |
| 7 | +For the MVP deployment path, tenant ingestion is **push-based**: |
| 8 | + |
| 9 | +1. The tenant workspace reads its own repository content. |
| 10 | +2. The tenant sends raw file deltas or `parsed-chunk-v1` records to the deployment. |
| 11 | +3. The KB server validates the payload, stamps the authoritative tenant tuple, embeds the chunk text server-side, and writes into the shared `knowledge-base` collection. |
| 12 | + |
| 13 | +The deployment does **not** need clone credentials for the MVP path. Server-side repo cloning is a later exploration owned by [#11731](https://github.com/neomjs/neo/issues/11731) and only starts if push-based ingestion proves insufficient. |
| 14 | + |
| 15 | +This model pairs with the D0 scheduler taxonomy in [#11721](https://github.com/neomjs/neo/issues/11721): local maintainer checkout sync stays local-only, while cloud tenant content arrives through the push-based path below. |
| 16 | + |
| 17 | +## Entry Points |
| 18 | + |
| 19 | +Use the same underlying ingestion service through two facades: |
| 20 | + |
| 21 | +| Facade | Use when | Volume / lifecycle | |
| 22 | +|---|---|---| |
| 23 | +| `ingest_source_files` | A tenant agent or push client sends a bounded incremental change set to the cloud MCP endpoint. | MCP-callable and volume-gated by `mcpSyncMaxChunks`; split or use the CLI when the gate refuses. | |
| 24 | +| `npm run ai:ingest-tenant -- <tenantId> ...` | A deployment operator, CI job, or onboarding script performs an initial import, full backfill, or large re-push. | Runs on the deployment host, bypasses the MCP turn-volume gate via `viaMcp: false`, and holds the heavy-maintenance lease. | |
| 25 | + |
| 26 | +Both facades call `KnowledgeBaseIngestionService.ingestSourceFiles()`. Do not document a third ingestion path for the MVP unless the implementation adds one. |
| 27 | + |
| 28 | +## Repository Identity |
| 29 | + |
| 30 | +Every pushed `parsed-chunk-v1` record belongs to this path-identity tuple: |
| 31 | + |
| 32 | +| Field | Operational rule | |
| 33 | +|---|---| |
| 34 | +| `tenantId` | Server-derived from the authenticated caller. A payload may carry a tenant claim, but it is not authoritative. | |
| 35 | +| `repoSlug` | Tenant-owned repository identifier. It is namespaced by `tenantId`, must be deterministic, and must never contain credentials. | |
| 36 | +| `rootKind` | Required repository topology hint: `neo-workspace`, `bare-repo`, or `external-source`. It selects hydration assumptions for content under the same `repoSlug`. | |
| 37 | +| `sourcePath` | Forward-slash-normalized path relative to the `repoSlug` root. It is never resolved against the KB server's `neoRootDir`. | |
| 38 | + |
| 39 | +`branch` is still useful operational metadata for the source branch or ref that |
| 40 | +produced a push, but it is part of the deployment runbook and tutorial evidence, |
| 41 | +not part of the current `parsed-chunk-v1` required schema. |
| 42 | + |
| 43 | +Recommended `repoSlug` shape: |
| 44 | + |
| 45 | +```text |
| 46 | +<provider-or-org>/<repo-name> |
| 47 | +``` |
| 48 | + |
| 49 | +Examples: |
| 50 | + |
| 51 | +```text |
| 52 | +acme/app |
| 53 | +acme/docs |
| 54 | +internal/platform |
| 55 | +``` |
| 56 | + |
| 57 | +If a tenant has multiple repos, each repo gets its own stable `repoSlug`. Manifests, tombstones, reconciliation, retention, alerting, telemetry, and source-family inventory remain scoped per `{tenantId, repoSlug}`. A bulk import that mixes repos must still let each record or batch resolve the correct `repoSlug`. |
| 58 | + |
| 59 | +Do not derive `repoSlug` from a credential-bearing remote URL. Normalize it from an explicit non-secret name chosen by the tenant or deployment operator. |
| 60 | + |
| 61 | +## Credential Boundary |
| 62 | + |
| 63 | +The push-based MVP path is credential-free from the KB server's perspective: |
| 64 | + |
| 65 | +- The tenant workspace already has access to its own repository. |
| 66 | +- The tenant push client reads local files and sends content or parsed chunks. |
| 67 | +- The KB server receives ingestion payloads, not Git credentials. |
| 68 | + |
| 69 | +Credential-bearing Git URLs are therefore rejected or treated as deferred clone-exploration input. They must not appear in: |
| 70 | + |
| 71 | +- `repoSlug` |
| 72 | +- logs |
| 73 | +- manifests |
| 74 | +- tutorial snippets |
| 75 | +- graph-visible configuration |
| 76 | +- source-family inventory output |
| 77 | + |
| 78 | +If a future server-side clone path becomes necessary, [#11731](https://github.com/neomjs/neo/issues/11731) owns the credential transport and storage contract before implementation begins. |
| 79 | + |
| 80 | +## Parser Dispatch |
| 81 | + |
| 82 | +The parser decision is per source family, not per tenant: |
| 83 | + |
| 84 | +| Source family | Default dispatch | |
| 85 | +|---|---| |
| 86 | +| Neo-supported text/source formats | Raw file delta to `ingest_source_files`; server-side parser or `raw-text` fallback. | |
| 87 | +| Custom but trusted operator-installed formats | Raw file delta with a registered `parserId`; server-side parser execution is operator-gated. | |
| 88 | +| Custom, untrusted, non-JS, or tenant-owned parser logic | Client-side parser emits `parsed-chunk-v1`; the KB server validates and embeds only the parsed records. | |
| 89 | +| Unknown format | Record as `unsupported` or `client-parser-required`; do not silently skip. | |
| 90 | + |
| 91 | +The KB server owns embeddings. `parsed-chunk-v1` records carrying an `embedding` field are rejected; pre-embedded records belong to restore-only backup paths, not ingestion. |
| 92 | + |
| 93 | +## Source-Family Inventory |
| 94 | + |
| 95 | +Before onboarding a tenant repository, produce a source-family inventory. The inventory is the handoff from Sub E into the day-0 tutorial work in [#11728](https://github.com/neomjs/neo/issues/11728). |
| 96 | + |
| 97 | +Use this checklist: |
| 98 | + |
| 99 | +| Source family | Questions to answer | |
| 100 | +|---|---| |
| 101 | +| Runtime source | Which languages and module systems are present? Which can use Neo-shipped parsers, and which require client-side parser output? | |
| 102 | +| Tests | Which unit, integration, e2e, fixture, and test-helper trees should be indexed? Which test artifacts should be excluded? | |
| 103 | +| Docs | Which Markdown, ADR, API, OpenAPI, generated-doc, and runbook files are authoritative? | |
| 104 | +| Config and deployment | Which package, Docker, CI, env-template, and infrastructure files should be indexed? Which carry secrets or local-only values and must be excluded or redacted? | |
| 105 | +| IDE/header/test-library equivalents | Which project-specific metadata files are needed for agents to understand conventions? | |
| 106 | +| Generated artifacts | Which files are generated and should be excluded unless they are the source of truth? | |
| 107 | +| Custom formats | Which formats need client-side parser output? Who owns parser versioning and deprecation? | |
| 108 | + |
| 109 | +Each inventory row should choose one dispatch outcome: |
| 110 | + |
| 111 | +```text |
| 112 | +server-raw |
| 113 | +server-parser:<parserId> |
| 114 | +client-parsed:<parserId> |
| 115 | +unsupported |
| 116 | +excluded |
| 117 | +``` |
| 118 | + |
| 119 | +## Deletion and Manifest Policy |
| 120 | + |
| 121 | +Incremental pushes should include deletion intent. Prefer this default shape: |
| 122 | + |
| 123 | +- `deleted` tombstones for explicit deletes. |
| 124 | +- `baseRevision` + `headRevision` when the push client can provide a reliable SHA range. |
| 125 | +- `manifestSnapshot` when the push point is meant to advance the claimed live file set for a repo. |
| 126 | + |
| 127 | +`manifestSnapshot.repoSlug` must match the repo whose `pathsAfterPush` it describes. A missing manifest does not authorize deleting earlier rows; it only means that push did not advance the claimed-state baseline. A bulk initial import can skip manifest state, but the deployment should follow it with a manifest-carrying push or an explicit claimed-state resync before relying on reconciliation to delete orphans. |
| 128 | + |
| 129 | +## Operational Flow |
| 130 | + |
| 131 | +1. Pick a stable `tenantId`, one or more secret-free `repoSlug` values, and the `rootKind` for each ingested source root. |
| 132 | +2. Build the source-family inventory. |
| 133 | +3. Choose dispatch for each family: raw server parse, registered server parser, client-side `parsed-chunk-v1`, unsupported, or excluded. |
| 134 | +4. Run initial import with `ai:ingest-tenant` when volume exceeds the MCP gate. |
| 135 | +5. Wire incremental `pre-push` or CI pushes through `ingest_source_files`. |
| 136 | +6. Include tombstones and revision boundaries; include manifests at reconciliation points. |
| 137 | +7. Fail the hook or CI job on structured ingestion errors instead of silently dropping files. |
| 138 | +8. Verify retrieval against the tenant corpus plus `neo-shared` content before handing the deployment to agents. |
| 139 | + |
| 140 | +## Evidence Boundary |
| 141 | + |
| 142 | +This guide is an L1 operational contract. It does not require new runtime behavior by itself. Add tests only when implementation touches a real seam, for example: |
| 143 | + |
| 144 | +- repoSlug normalization or rejection logic; |
| 145 | +- credential-bearing URL redaction/rejection; |
| 146 | +- parser-dispatch branching; |
| 147 | +- manifest/tombstone handling; |
| 148 | +- tutorial fixture executability. |
| 149 | + |
| 150 | +The day-0 tutorial should reuse this model rather than redefine it. |
| 151 | + |
| 152 | +## Related |
| 153 | + |
| 154 | +- [Hook Wiring](./HookWiring.md) — the `ingest_source_files` and `ai:ingest-tenant` facades. |
| 155 | +- [Custom Parsers](./CustomParsers.md) — `parsed-chunk-v1` and parser execution boundaries. |
| 156 | +- [Custom Sources](./CustomSources.md) — full-corpus Source path, mostly not the push-based tenant default. |
| 157 | +- [Security](./Security.md) — tenant stamping, spoof rejection, parser trust, and KB-as-cache recovery. |
| 158 | +- [#11721](https://github.com/neomjs/neo/issues/11721) — D0 scheduler taxonomy that separates local-only maintainer sync from cloud tenant ingestion. |
| 159 | +- [`identity-tuple.md`](../../../ai/services/knowledge-base/parser/identity-tuple.md) — authoritative path identity tuple. |
| 160 | +- [`deletion-signaling-contract.md`](../../../ai/services/knowledge-base/parser/deletion-signaling-contract.md) — tombstone, manifest, and revision-boundary mechanics. |
0 commit comments