feat(sync): observable federation sync state with live progress UI#9
Merged
Conversation
…tion The home/list page never set a breadcrumb, so an integrating shell (data-fair) was left with an empty trail when the Registry tab opens. Forward an artefact count breadcrumb, mirroring the processings/catalogs list pages. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Design for making remote-registry sync observable: lock-derived running state, persisted syncProgress, websocket push instead of polling, and 409 on a concurrent manual sync. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Nine TDD tasks: pure derivation helpers, syncProgress schema, ws boot, syncState on reads, 409 on concurrent sync, progress persist+emit, UI composable, live detail panel, list chip. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@data-fair/lib-express/ws-server.js requires ws@8, and @data-fair/lib-vue/ws.js requires reconnecting-websocket@4. Both are declared as peerDependencies by those packages and were unmet. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rdering - startSync's background .finally() now catches locks.release() failures too, not just runSync's own errors -- an unguarded Mongo failure there would otherwise become an unhandled promise rejection and kill the process. - Strengthen the "unknown registry is still 404, not 409" test with a deterministic pre-held-lock guard so it actually fails when the route's 404 check and lock-acquire are reordered (the previously-suggested post-hoc lock-absence check races the background release and doesn't reliably catch this). Adds a dev-only GET /api/test-env/locks/:id route and a syncLockExists() test helper as secondary sanity tooling. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ct sync loop The final runSync update re-set syncProgress.done even though the loop (or the pre-loop write, for zero artefacts) already writes it, which made it impossible for a test to detect a regression in the loop's own write. Remove the redundant set, and add a test that selects one artefact against a non-resolving remote so the loop body actually runs, covering both the per-artefact progress write and the per-artefact error path.
The vite dev server had pre-bundled its deps before reconnecting-websocket was installed, so it could not regenerate this until it restarted. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…gger The optimistic syncState='running' write in syncAction left syncProgress untouched, so a registry with no prior sync showed a disabled button over an empty card body, and a registry with a completed previous sync briefly showed that run's finished progress bar. Clear syncProgress alongside the optimistic state write, and add a syncRunning-only template branch (spinner + "starting" caption) for the window before the first websocket event lands.
a61123d made the artefacts list emit a single artefact-count breadcrumb so an embedding shell does not open the Registry tab on an empty trail. The layout spec still asserted zero breadcrumbs on the home page and had been failing since. Assert the intended crumb instead. Unrelated to the federation sync work on this branch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A second real registry process as the upstream (no api code changes: PORT, MONGO_URL and DATA_DIR are already env-mapped). Extends dev:fixtures to wire the mirror, and closes the gap where syncNpmArtefact/syncFileArtefact have never run under test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Five tasks: upstream process (env, scripts, zellij, status), test-support axios helpers, fixtures seeding + mirror wiring, an end-to-end sync spec that mirrors real bytes, and CI + docs. Includes a human gate: an agent may not start the upstream dev process. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sync mirrors from an upstream registry, so exercising it needs two. Pointing a registry at itself cannot work: selecting an artefact that already exists locally without an `origin` is rejected with 409. No api code changes — PORT, MONGO_URL and DATA_DIR are already mapped in custom-environment-variables.js, and node-config gives env vars precedence. A separate DATA_DIR matters: filesStorage.clean() does `rm -rf $dataDir/tarballs`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
upstreamBaseURL is a function, not a const: this module is imported by every spec, and a module-level throw on a missing DEV_UPSTREAM_API_PORT would break all of them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Seeds the upstream registry, mints a read key owned by org test1 (which needs an access grant first), registers the mirror downstream and selects two artefacts with ids disjoint from the local fixtures — reusing a local id hits the 409 that also makes self-mirroring impossible. Stops short of syncing: the "Sync now" button is the point. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The first test anywhere to execute syncNpmArtefact/syncFileArtefact. Until now the only passing sync in the suite mirrored zero artefacts, and the one-artefact test asserted a DNS failure — so downloading a tarball, writing it into files-storage and setting `origin` had never run. Mutation-verified: removing the writeStream in syncNpmArtefact makes the byte-identical assertion fail. Also the first time the mirrored-artefact 403 guards are exercised against an artefact that really was mirrored, rather than one whose origin was forced by the test-env router. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Also adds a readiness wait for both APIs — the existing "Start dev API" step had none and merely got away with racing the test run. AGENTS.md: document the api-upstream pane, and correct the per-file test command (`npm run test <file>` is a compound `&&` script, so the file argument only reaches the e2e project). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The api test suite calls cleanUpstream() between runs, which deletes the upstream's api keys — including the read key dev:fixtures had minted. Re-running fixtures did not recover: the read-key step skipped on the local output file alone, and the registry step skipped on 409, so the downstream kept ciphering a key the upstream no longer had. Browsing the mirror then 401'd. Two fixes: - Consult the upstream (as the owner) before skipping the read key. The read-key LIST endpoint filters by the caller's account, so a key owned by org test1 is only visible through a test1-scoped session, never a plain superadmin one — querying as superadmin always returned empty and re-minted every run. - Always PATCH the registry's apiKey to the current value, so a re-minted key propagates to the stored ciphered key instead of leaving a stale one. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
# Conflicts: # AGENTS.md # api/package.json # api/src/app.ts # package-lock.json
The installer stage's `npm install -w ui` rewrites the lock file against fresh registry metadata: it can bump an optional transitive edge (here @emnapi/core -> @emnapi/wasi-threads@1.2.2) without re-locking the entry, leaving the in-image lock out of sync. The api-installer stage's `npm ci -w api` then fails with EUSAGE. Restore the committed lock first so npm ci validates what the repo actually pins. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make federation sync observable and safe to operate: remote-registry reads expose a derived
syncState(running/interrupted/idle), sync progress is persisted and streamed over websocket to the admin UI, and a manual sync against a registry that is already syncing is rejected with 409. A second registry process (api-upstream) is wired into dev and CI so the mirror path is exercised against a real upstream.What changed
api: puresyncStatederivation from the lock +syncProgress.startedAtvslastSyncAt(a stored "running" flag would lie after a crash);syncProgresspersisted per artefact and published on a per-registry ws channel; ws pub/sub server booted with admin-only subscriptions;POST /:id/syncreturns 409 when the lock is held.ui: live progress bar and current-artefact display on the registry page via auseRegistrySynccomposable (reconnecting-websocket), sync-state chips on the registries list, Sync button disabled while running, friendly 409 handling.api-upstreamzellij pane + npm script,dev:fixturesseeds the upstream and registers the mirror, CI runs the upstream registry alongside the api tests, newtests/remote-registries-sync.api.spec.tsmirrors real npm and file artefacts end to end.Dockerfile: restore the pristine lockfile in theapi-installerstage — theinstallerstage'snpm install -w uirewrites the in-image lock against fresh registry metadata (bumping an optional@emnapi/*edge without re-locking it), and the subsequentnpm ci -w apirejected it as out of sync. This was breaking the image build on main too.Why: sync used to be a fire-and-forget background job — no way to tell whether it was running, had died mid-way, or what it was doing; and nothing in dev or CI ever executed the real mirror path.
Regression risks:
POST /remote-registries/:id/syncpreviously always answered 202 (a held lock was silently skipped); it now answers 409. Any automation that fires this endpoint blindly must tolerate 409 — the dailysyncAllRemoteRegistriesjob treats it as normal replica contention.ws-server+ws-emitter, newwsdependency) now starts on every boot, in production too. Subscriptions are refused for everyone except admin-mode sessions (canSubscribereturns false; ws-server short-circuits for admins), but it is a new listening surface on the existing HTTP server.syncRemoteRegistry()changed semantics: it now returnsfalseinstead of silently no-op'ing when the lock is held, and sharesrunSyncwith the new fire-and-forgetstartSync(). Internal callers were updated; nothing external imports it.syncProgress; reads (list + detail) now carrysyncState/syncProgress. Additive, admin-only endpoints.