Seeding

Seeding

How the Cosmos containers are filled from Track and Eagle. The merged registry those sources produce is described in Architecture; which id means what is Identity-Model.

Running a seed

src/seed/sources.js holds all I/O, src/seed/transform.js is pure, and src/scripts/seed-nosql.js orchestrates.

node src/scripts/seed-nosql.js [--live] [--only projects,documents,boundaries]
                               [--limit-documents N] [--reconcile] [--max-surplus N]

Dry run by default; --live is required to write. The gates run in both modes, so a dry run is a real pre-flight check.

Order is forced. Projects go first because every other container partitions by a canonical project id that only the merged registry can supply. --only still builds the registry even when projects are not written, since a stale index would misfile documents into the wrong partition.

Cosmos is behind a private endpoint, so a full seed runs on the devbox VM, detached with a log file, since a 60k-document run outlasts a single run-command invocation. Reaching the devbox: Operating-DEMI-Data-Plane#reach-the-data-plane.

Seeding the public config document

GET /config/public serves eagle-public's runtime configuration from a second item, id: 'public', in the config container. It is filled by its own script, separate from seed-nosql.js:

npm run db:seed-public-config             # dry run, prints what would be written
npm run db:seed-public-config -- --live   # write it
npm run db:seed-public-config -- --live --force   # overwrite an existing document

It reads eagle-api's live GET /api/config (the URL defaults per ENVIRONMENT, overridable with EAGLE_CONFIG_URL), keeps only the keys PUBLIC_KEYS allowlists (src/controllers/config.js), and upserts them as the public item. Parity with eagle-api comes from reading its live answer, not from a hand-kept key list. A malformed EXTENDED_PROJECT_PAGES is dropped with a warning.

Runs on the devbox, same as every other Cosmos script: demi-cosmos-* is private-endpoint-only.

This is the bootstrap, not the sync. eagle-api pushes every /api/config payload change to PUT /eagle/config/public on its own, so a hand edit to eagle-api's Mongo Config document reaches this document on the next read of /api/config — see ADR-012-Public-Config-Surface. Run this script to create the public item before the first push, or to force it back to eagle-api's live payload.

Reconciling upstream deletes

--reconcile also deletes the rows the fetch did not produce, through the same helpers the DELETE controllers use. That is how a hard delete in Eagle is noticed. It needs both the projects and documents stages, refuses to run with --limit-documents or without COSMOS_ENDPOINT, and stops before deleting anything when one container would lose more than the surplus ceiling — max(50, 2% of the rows, --max-surplus N). Removed rows are written to a reconcile-<timestamp>.ndjson log.

Source facts the transform encodes

Finding	Consequence
s3Key is null on every Eagle document; internalURL holds the key	Reading s3Key would seed the whole corpus with no downloadable file
isPublished is true on only 66% of documents that are unambiguously public by read[]	isPublished is derived from read[], never copied. Copying it would hide a third of the corpus
internalSize is a number or a numeric string	Coerced through toNumber
contentExtracted is true on 99% upstream, but a fresh DEMI has no chunk data	Reset to false
A small share of documents reference a project absent from the public list	Dropped and counted; a silent drop looks like a complete corpus
Eagle's API caps pageSize at 100 regardless of what is requested	Asking for 1000 silently reads a tenth of the data while appearing to work
Eagle type, milestone and projectPhase are ObjectId refs into a 213-item List	Resolved to labels at seed time; an unresolvable ref keeps its raw value rather than becoming null

Streaming

Documents stream rather than accumulate. fetchAllPages({accumulate: false}) never builds the array, and the orchestrator buffers per project and flushes at FLUSH_THRESHOLD = 100, the Cosmos bulk limit, so a full buffer is exactly one request. Peak RSS is about 123 MB and flat. The page handler is awaited; without that the flush-per-page backpressure disappears and memory grows unbounded, which a test asserts.

Progress reports per page, not per dataset. A per-dataset callback emits nothing for about twenty minutes, which is indistinguishable from a hung process.

Safety gates

• fetchAllPages throws on a short count. A mid-run upstream hiccup returning a partial page would otherwise read as end-of-data and quietly seed tens of thousands fewer documents.
• An unexpected response envelope throws rather than reading as zero results and seeding an empty database.
• The seed exits non-zero on: a synthetic trackProjectId >= 8,000,000, more than 20 duplicate project names, duplicate ids, any item with no read[], any item with no partition key, isPublished out of step with read[], and any document that is public under a non-public project.
• items.bulk is chunked at 100 inside src/db/cosmos-nosql.js rather than at the call sites. Cosmos rejects more, and a caller that forgot would fail only on the large projects.

Projects are never created from a compliance or notification feed. Only rows whose parent reference resolves to a project already in the registry are ingested; the synthetic-id and duplicate-name gates are what make the alternative fail loudly instead of filling the registry with unprovenanced rows.

See also

• Architecture — the merged registry, containers and authorization
• Sync-Architecture — how records arrive between seeds, and what a visibility change cascades
• Environment-Reality-and-Operational-Gotchas — running against a private-endpoint environment