Skip to content

feat(api): hot-reload webhook for the public data branch#70

Merged
themightychris merged 4 commits into
mainfrom
feat/hot-reload-webhook
May 19, 2026
Merged

feat(api): hot-reload webhook for the public data branch#70
themightychris merged 4 commits into
mainfrom
feat/hot-reload-webhook

Conversation

@themightychris
Copy link
Copy Markdown
Member

Closes #65. Builds on #66 (PR #68) — the data-repo reconciliation moved into the Node process, which this PR consumes via fastify.reconcileDataRepo.

Summary

  • POST /api/_internal/reload-data — hidden from the OpenAPI doc, bearer-auth via CFP_DATA_RELOAD_SECRET, constant-time compare, 503 when the secret is unset
  • Cheap pre-check via git merge-base --is-ancestor short-circuits the self-trigger case (push-daemon's own commits arriving via the webhook) without acquiring the data-repo lock
  • After a reconcile, outcome === 'in-sync' short-circuits the rebuild; otherwise the in-memory state + FTS index are rebuilt in place against the post-fast-forward tree
  • Plan in plans/hot-reload-webhook.md; spec section in specs/behaviors/storage.md; operator notes in docs/operations/runbook.md

Decisions worth flagging

  • The gitsheets Store caches a dataTree per Sheet. A fast-forward merge updates the working tree but the cached snapshot stays stale, so queryAll() returns the pre-merge records. The reload helper re-opens the public store and Store.swapPublics it. The transact path is unaffected — repo.transact builds a fresh workspace from the parent commit each call.
  • In-place mutation of fastify.inMemoryState. Service classes captured the state object at boot; replacing it would orphan them. The helper builds a fresh state to a local variable, then clear() + set() every Map in a tight synchronous block on the live object. Failure during the build leaves the running state untouched; failure during the FTS reload is loud and the route 5xx's so the operator restarts.
  • 503 vs. 401 ordering. A missing/empty Authorization header gets 401 before the route checks whether the secret is configured. This way unauthenticated probes can't tell whether the env var is set; only callers that pass some bearer token receive a 503-vs-401 distinction (and a sealed-secret operator can't pose as an unauthenticated probe).

Operator follow-up

This PR intentionally doesn't include the GitHub Actions workflow that calls the webhook — that lives on the codeforphilly-data repo. Drop the following into codeforphilly-data/.github/workflows/notify-deployments.yml:

```yaml
name: Notify deployments
on:
push:
branches: [published]

jobs:
notify:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
target:
- { name: sandbox, url: 'https://next-v2.codeforphilly.org/api/_internal/reload-data' }
# - { name: prod, url: '...' } when prod stands up
steps:
- name: Trigger reload on ${{ matrix.target.name }}
env:
CFP_DATA_RELOAD_SECRET: ${{ secrets.CFP_DATA_RELOAD_SECRET }}
run: |
curl --fail-with-body -sS -X POST "${{ matrix.target.url }}"
-H "Authorization: Bearer $CFP_DATA_RELOAD_SECRET"
-H "Content-Type: application/json"
-d "$(jq -nc --arg sha "${{ github.sha }}" '{branch:"published", commitHash:$sha}')"
```

The CFP_DATA_RELOAD_SECRET repository (or environment) secret on codeforphilly-data must match the cluster-side sealed Secret. Per environment, the sealed Secret lives in cfp-sandbox-cluster/codeforphilly-ng.secrets/ (or production equivalent) — not in this repo. Generate with openssl rand -base64 48, then seal into the GitOps repo and mirror the same plaintext into the data repo's secret.

Test plan

  • npm run -w apps/api type-check — clean
  • npm run -w apps/api test — 240 tests pass (including 7 new in internal-reload.test.ts)
  • npm run lint — clean
  • Sandbox cluster: set CFP_DATA_RELOAD_SECRET, mirror it into the codeforphilly-data secret, push a no-op commit on published, verify the workflow's curl logs the 200 noChanges response and the API logs hot-reload short-circuit: commit already in local HEAD.
  • Sandbox cluster: import + merge a legacy-importpublished change, verify the workflow's curl logs the 200 outcome: fast-forwarded response and the new records are visible immediately without a pod restart.

themightychris added a commit that referenced this pull request May 19, 2026
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
themightychris and others added 4 commits May 19, 2026 09:57
Captures the contract for POST /api/_internal/reload-data: bearer-auth,
optional body, cheap ancestry pre-check, lock-protected reconcile, in-place
state rebuild. Plan tracks #65.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
POST /api/_internal/reload-data — closes the loop on #65. Hidden from
the OpenAPI doc; bearer-auth via CFP_DATA_RELOAD_SECRET with constant-
time compare; 503 when the secret is unset so the deployment surface
stays uniform.

Two layers of no-op coverage:
  1. Cheap pre-check via `git merge-base --is-ancestor` — handles the
     self-trigger from push-daemon-emitted pushes without acquiring the
     data-repo lock or hitting the network.
  2. After a reconcile, outcome === 'in-sync' short-circuits the rebuild.

Otherwise the reconcile state machine (#66) runs under the data-repo
lock and the helper at `store/memory/reload.ts` re-opens the public
store (gitsheets caches a dataTree per Sheet, so the snapshot has to be
replaced after a fast-forward), builds a fresh InMemoryState, mutates
the live Maps in place, swaps the public-store reference, reloads the
FTS index in a single SQLite transaction, and invalidates the facet
cache. If the rebuild throws after the swap has begun, the route logs
loudly and returns 500 so the operator knows a restart is warranted.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds CFP_DATA_RELOAD_SECRET to the env-var table in deploy.md and a
"Hot-reload webhook" section to runbook.md covering manual triggering
and the response shapes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@themightychris themightychris force-pushed the feat/hot-reload-webhook branch from 91333f4 to 10abee7 Compare May 19, 2026 13:57
@themightychris themightychris merged commit 26571fc into main May 19, 2026
1 check passed
@themightychris themightychris deleted the feat/hot-reload-webhook branch May 19, 2026 14:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Webhook endpoint for hot-reload on push to the published data branch

1 participant