ENGAGE Rows Reference

ENGAGE Rows Reference

Comment periods that ENGAGE owns: who owns which field, how a push finds its row, what a delete does, and what the two reconcile scripts do with them. Why the flow exists: ADR-014-Sync-Out. How the queue and worker run: Sync-Architecture. The ingest request: Connecting-an-Application-to-DEMI.

An ENGAGE-owned row is a commentPeriods row with sourceSystem: 'engage' and an engagementId. It is written by PUT /api/engage/engagements/{engagementId} (src/controllers/nosql/engage-comment-period.js) and by the Eagle mirror (src/controllers/nosql/comment-period.js).

Field ownership

Owner	Fields
ENGAGE	projectId, engagementId, eagleProjectId, isMet, metURL, metURLAdmin, metBannerImageUrl, informationLabel, instructions, dateStarted, dateCompleted, dateAdded (set once, on create), isDeleted, isPublished, read, sources.engage, engagePushedAt, syncVersion, movedFromProjectId
Eagle	eagleId (after the first Eagle push), sources.eagle, eaglePushedAt, and the staff fields the Eagle mirror copies (staffFields in comment-period.js), except metURLAdmin
Sync-out	syncOut (per consumer: sentVersion, sentAt, status, error, projectId sent under, failedVersion after a failure, candidateEagleId on a conflict, droppedIds: the last 20 old Eagle ids a project move deleted, and a short-lived create claim), and eagleId once the consumer creates the Eagle period

Rules that follow:

• An Eagle push to an ENGAGE-owned row writes only the Eagle column. It never changes dates, URLs, labels, read, isPublished or isDeleted.
• read comes from engagement.isPublished: public, staff, sysadmin when true, staff, sysadmin when false. It is then capped by the parent project's read[]. isPublished is true when the stored read holds public. A notification parent is refused with PARENT_KIND_UNSUPPORTED; those engagements go to Eagle from ENGAGE directly.
• start and end arrive as YYYY-MM-DD HH:MM:SS with no zone. DEMI reads them as UTC and stores dateStarted and dateCompleted as ISO 8601 with Z. ISO 8601 is also taken, a time with no zone read as UTC. Any other text is refused with PUSHED_DATES_INVALID.
• Each source orders its own pushes. ENGAGE pushes are compared on engagePushedAt and Eagle pushes on eaglePushedAt, so one source's clock never refuses the other's push.
• Every ENGAGE write adds 1 to syncVersion, even when no field changed. Sync-out compares it with what it last sent. engagePushedAt cannot serve, because a slug-only edit in ENGAGE leaves ENGAGE's updated_date as it was.
• A comment period whose read changed level re-derives the read of its comments.

Finding the row

An ENGAGE push writes to the first of these that exists:

1. the row whose engagementId matches the path, in the parent's partition first, then in any other;
2. the row whose id equals engagement.trackingId (an Eagle period ENGAGE already pushed, adopted on the first push);
3. the row engage-<engagementId>, created when none exists.

A row id never changes after creation. If the row found already carries a different engagementId, the push is refused with TRACKING_ID_CLAIMED.

A row found under another project has moved in ENGAGE. DEMI writes it under the new project, sets movedFromProjectId, then deletes the old copy. A failed delete is logged; the marker lets a lookup read the two copies as one move, not a duplicate.

An Eagle push (PUT /api/eagle/commentperiods/{eagleId}) writes to the first of these:

1. the row stored under the Eagle id;
2. a row whose eagleId equals it, in the same partition first, then in any other;
3. an ENGAGE-owned row on the same metURL that has no eagleId yet. A delete never joins this way.

This is the id miss case: the echo of a period that sync-out has just created can reach DEMI before sync-out stores the created period's eagleId. Join 3 ties the echo to the ENGAGE row. A row that already has another eagleId is never taken. With no match, the Eagle push creates an ordinary Eagle row, except a delete: a delete for a period DEMI holds no row for writes a staff-only tombstone and answers 200. It writes nothing, and answers 200 with action: 'ignored', only when an ENGAGE row lists that Eagle id in syncOut.eagle.droppedIds. Sync-out's project-move path adds the old Eagle id there before it sends the DELETE, so the echo of that delete is ignored.

Join 2 across partitions covers a project move: the row sits under the new project while Eagle still files the period under the old one. The Eagle-owned fields are written where the row is.

A delete of an Eagle id that the row no longer names changes nothing. It is the echo of sync-out removing the old period after a project move, and must not tie the row back to it.

A comment push whose period is stored as engage-<engagementId> finds the period by its eagleId. Only an ENGAGE-owned row that a system read can see is taken this way, so a sealed or read-less period still refuses its comments with 404. An eagleId stored on two period rows is a 500.

Deletes

ENGAGE sends engagement.isDeleted: true. DEMI keeps the row as a tombstone: isDeleted is true and read is capped to level 2 (staff). The comments under it follow. sync-out sends DELETE to eagle-api when the row has an eagleId, and sends nothing when it has none. A later Eagle push cannot clear the flag.

What reconcile-eagle excludes

reconcile-eagle.js leaves ENGAGE-owned rows out of its comment period diff and out of --drop-orphans, which deletes periods whose ENGAGE slug no longer resolves. The report counts them as engageOwned. Drift classes: Eagle-Mirror-Reference#drift-classes.

reconcile-engage

node src/scripts/reconcile-engage.js [--repair] [--store] [--limit N]

It compares every ENGAGE-owned row with Eagle's copy and with ENGAGE's public engagement list. Without --repair it writes nothing. --store saves the report under the cache id reconcile-engage-report; GET /api/admin/reconcile-engage (admin) returns it. --limit caps the rows checked against Eagle.

Report field	Meaning
neverSynced	A live row with no eagleId, other than the skipped ones.
skipped	A live row with no eagleId that sync-out will not create in Eagle: it has no metURL, or sync-out recorded skipped at its current syncVersion.
missingInEagle	The row has an eagleId and Eagle holds no such period.
drift	Eagle's copy differs in dateStarted, dateCompleted (compared to the minute), isPublished or metURL; or the row is deleted and Eagle still holds the period (isDeleted).
duplicates	An Eagle project with more than one period on the same metURL.
conflicts	Sync-out found an Eagle period on the row's metURL that DEMI already holds as an Eagle-owned row, and did not adopt it. Carries candidateEagleId.
engageMissingInDemi	ENGAGE lists an engagement under a project and DEMI has no row for it. Engagements under a notification are skipped.
queued	Rows --repair queued.
warnings	Checks that could not run, such as an unset setting or an HTTP error.

--repair clears the row's sent version and queues the neverSynced, missingInEagle and drift rows again, so DEMI's copy wins. Its messages carry repair: true, so the worker never drops one as an already-failed redelivery. It queues nothing when SYNC_OUT_EAGLE_ENABLED is not true. The script never writes to Eagle itself. skipped, conflicts, engageMissingInDemi and duplicates are report only.

ENGAGE's public list shows published engagements only and leaves out internal ones, so a DEMI row missing from it is not checked. Metadata reads that fail are one warning per HTTP status, listing the engagement ids. The ENGAGE side is skipped when ENGAGE_API_BASE is unset. The Eagle side is skipped when the sync-out Eagle settings are unset.

The nightly timer reconcileEngage runs with --store and never with --repair. It exists only when RECONCILE_ENGAGE_SCHEDULE is set.

Settings

Setting	Meaning
SYNC_OUT_QUEUE	Queue name. Empty: no worker is registered and enqueue fails.
SYNC_OUT_EAGLE_ENABLED	true turns the Eagle consumer on. Anything else queues nothing for it.
EAGLE_PROTECTED_API_BASE	eagle-api's authenticated /api base. Not EAGLE_API_BASE, which is the public base the seed reads.
EAGLE_KC_CLIENT_ID, EAGLE_KC_CLIENT_SECRET	Client-credentials login at the realm KEYCLOAK_URL and KEYCLOAK_REALM name. The client id is demi-sync-out; the secret is a Key Vault reference.
EAGLE_ENGAGE_MILESTONE	Milestone id sent on every Eagle write. eagle-api stores an invalid id when it is missing.
ENGAGE_API_BASE	ENGAGE's API, read by both reconcile scripts. Bicep param engageApiBase: test https://epic-engage-web-test.apps.gold.devops.gov.bc.ca/api, prod https://epic-engage-web-prod.apps.gold.devops.gov.bc.ca/api.
RECONCILE_ENGAGE_SCHEDULE	NCRONTAB schedule of the reconcileEngage timer.
DEMI_ENGAGE_PRINCIPALS	Registry principals allowed to call the ingest. Bicep param engagePrincipals, default apim:engage. Empty refuses everyone.