Eagle DEMI Re Push And Backfill

Eagle DEMI Re-Push and Backfill

How to resend Eagle records to DEMI, undo the retired staff read rule on stored rows, and check DEMI against eagle-api. Background: Eagle DEMI.

Re-push records from eagle-api

The push sweep re-sends rows flagged demiPushPending on its own (see Eagle DEMI Push Sweep). Use this script for rows with no flag, such as a backfill or a DEMI reseed. The script does not touch the pending fields: a flagged row it re-sends stays flagged until the sweep clears it, and a failed re-push does not flag the row.

A DEMI-side narrowing or takedown of a project or document survives a re-push (levelHeldAt). Other kinds are not held: remove their taken-down and narrowed ids from the id file. Projects and documents go by --ids-file, because of volume. A live project run requires it.

Kinds are singular camelCase: project, document, commentPeriod, comment, organization, projectNotification, recentActivity, user, group, inspection, inspectionElement, inspectionItem.

1. Open a shell in an eagle-api pod for the target environment. On prod, set LOG_LEVEL=info in that shell. The prod pod runs at error, which hides the progress lines.

2. Dry run (the default) and read the counts:

   node scripts/demi-repush.js --kind commentPeriod

3. Run it live:

   node scripts/demi-repush.js --kind commentPeriod --live --state /tmp/repush.json

   For projects or documents, add --ids-file <file>. For projects, add --concurrency 1 too.

Flag	Use
--kind <name>, --kinds <a,b>	One kind, or several run in order
--ids-file <path>	Only these 24-character ids
--live	Push. Without it nothing is pushed
--concurrency N	Pushes in flight. Default 4
--rate N	HTTP calls a minute. Default 150. The subscription allows 300, shared with the live pods
--state <path>	Checkpoint file. A rerun resumes after it
--continue-on-failure	With several kinds, go on after a kind had failures

Exit codes: 0 all pushed, 1 run failed, 2 bad arguments or DEMI not configured, 3 DEMI refused some records.

The --state file records lastId, the last record read, whether it landed or failed. A rerun with the same --state starts after it. Ids DEMI refused collect in failedIds, and a resume skips them. To retry them, put them in an --ids-file and give a separate --state path: a run over a different id list starts its state file over.

The script reads a kind through a cursor, 100 records per batch. If Mongo drops the cursor (CursorNotFound), the script reopens it after the last _id read, up to 3 times in a row.

Reverse the staff read rule on stored rows

backfill-eagle-ladder.js --reverse rewrites rows that carry the staff token the retired Eagle-mirror rule added. It changes a row only when the stored read[] is exactly what that rule and the old cap give for the same Eagle read and parent. Held rows, sealed rows and rows that differ are left alone.

The script exits 2 and writes nothing while src/helpers/eagle-acl.js in the build still exports withEagleStaff or seedAcl.

1. Deploy a DEMI build without that rule.

2. Dry run on the devbox and read the per-container counts:

   scripts/demi-devbox.sh run --env test -- 'git pull && yarn install && node src/scripts/backfill-eagle-ladder.js --reverse'

3. Run it live:

   scripts/demi-devbox.sh run --env test -- 'git pull && yarn install && node src/scripts/backfill-eagle-ladder.js --reverse --live'

4. If a container reports stale, run step 3 again. A failed count makes the script exit 1.

Check parity with eagle-api

src/scripts/parity-eagle.js replays eagle-api reads against eagle-api and DEMI with one identity and diffs the rows. It needs only a bearer token, read from the variable --token-env names.

Flag	Use
--identity	anonymous, staff or sysadmin
--token-env <VAR>	Variable holding the bearer token
--known-ids <file>	Ids already explained. Repeatable
--emit-ids <file>	Write the ids of rows in a known class
--only <read>	One read
--max-pages <n>	Stop after n pages
--download-sample <n>	Compare n downloaded files

src/scripts/parity-map.js groups known differences into the classes schema-default-false, seeded-from-prod, ladder-above-public, display-name-from-file-name and orphan-parent-missing-in-eagle. Any other difference counts as unexplained and exits 1.