Repository navigation
Proposal 0006
| Status | Unfinished Draft |
| Author | Fabian Kloosterman |
| Date | 2026-08-07 |
| Repository | ridescoredc-models |
| Decides | how packages and bundles reach a serving database, and what runs when |
| Depends on | proposal 0001 (packages and load), proposal 0003 (the presentation), proposal 0005 (what is loaded where) |
Delivery is pull-based. CI builds and signs a pipeline image and publishes data packages to a registry; each server runs a timer, pulls what it is entitled to, verifies the signature, and applies it. Nothing outside a server holds a credential that reaches it. Applying is split in two: generate turns a package, a presentation and the server's own addresses into serving SQL and a manifest, using no database and no credentials, so its output can be read and diffed first; apply loads the package, applies the serving SQL, then publishes the manifest. Promotion to production is a maintainer retagging a package in the registry.
There is no deployment workflow for data. A person runs the notebook, saves its
output, and loads that output into the database by hand with ogr2ogr. Nothing
records what was loaded or when, and there is no artifact to go back to.
Proposal 0001 replaces the notebook with a library that produces a versioned data
package and a load command. It says nothing about who runs that library, on
what machine, on what schedule, or how a package gets from a run into a database
that serves the map. This proposal covers that.
The website's own release process is a separate proposal. The two meet in one place: the order in which a data change and a website change are released (see §4.3).
- Data is produced and loaded by hand, so a refresh depends on somebody remembering the steps, and cannot be repeated or reviewed.
- Nothing records which data a database holds, so nobody can say what an environment is serving, or whether dev and production hold the same thing.
- There is no rollback. Recovering from a bad load means re-importing an earlier output, if somebody kept one.
- Nothing can be checked before it is applied — not the serving SQL, not the manifest, not whether a presentation matches the package it will be served with.
- Where the pipeline runs is undecided. It is a library and a CLI; nothing says who invokes it, when, or on what machine.
- Deployment credentials point the wrong way. Today a merge causes CI to connect to a server over SSH, so a system outside the server holds a key into it.
A deployment consumes two published artifacts, produced elsewhere:
| Holds | Built by | |
|---|---|---|
| the data package | datasets, their descriptions, and how the run was made | the pipeline — proposal 0001 |
| the deployment bundle | the serving SQL for each layer, and the manifest with endpoint addresses left open |
ridescore generate, from a package and the presentation — proposal 0003
|
The data package carries no presentation-derived material — no serving SQL, no manifest, nothing about palettes or labels. It is a claim about the world, and correcting a legend must never require republishing it.
A bundle declares which packages it applies to: a shape and a description revision, any data build (proposal 0003 §4.7). A routine refresh therefore publishes a package and reuses the bundle already deployed; a legend fix publishes a bundle over the package already deployed.
Apply is the only phase that changes anything:
load the package → apply the bundle's serving SQL
→ substitute this server's addresses
→ publish the manifest
The manifest is published last, so the browser learns about a layer only once the endpoint behind it exists. Nothing has to happen simultaneously, which cannot be guaranteed — only in order, which can.
Apply verifies the bundle's contract: a server refuses a package whose shape or description revision differs from what the bundle declares.
The only thing decided per server is its endpoint addresses. Everything else was settled when the bundle was built, which is what makes promotion mean something — production applies the same serving SQL that was examined on dev, rather than regenerating it from whatever image it happens to be running.
Nothing pushes into a server. CI builds and signs artifacts and publishes them to a registry; each server decides what to take.
CI ──build, sign, publish──► registry ◄──pull on a timer── dev server
▲ prod server
│
maintainer retags a package
to promote it
| the pipeline image | built and signed by CI on merge, pulled by each server. Shipping an image means the geospatial dependencies are built once and every server runs what was tested |
| a data package | published to the same registry, addressable by digest, pulled by whichever server is entitled to it |
| a deployment bundle | pulled alongside the package; a server checks that the package it holds matches the shape and description the bundle declares |
| what stays on a machine | its schedule, the channel it subscribes to, its resource limits — installed once by hand |
Dev pulls any current package; production pulls only promoted ones. A server verifies the signature and the identity of the workflow that produced a package before applying it.
Three things follow. No credential outside a server reaches it, so a compromised CI account cannot write to production. A server that was unreachable catches up on its next tick rather than missing a release. And promotion becomes a registry operation rather than an act performed on a machine.
A control endpoint allows "pull now" where waiting for the timer is not wanted.
| Push — CI connects to the server | Pull — the server fetches (proposed) | |
|---|---|---|
| credentials | a system outside the server holds a key into it; the blast radius of a compromised CI account includes production | the server holds a read credential for the registry, and nothing reaches inward |
| a server that is down | misses the release; somebody has to notice and re-run | takes it on the next tick |
| adding a server | give CI another key, and another job | subscribe it to a channel |
| latency | immediate | up to one poll interval, or immediate via the control endpoint |
| approval gate | a pause in a workflow: visible and auditable | a tag the server checks for — the gate lives in the registry rather than in the run |
| "deploy exactly this, now" | one click | wait for the poll, or use the control endpoint |
| what is deployed | whatever the job pushed, recorded in the job's log | whatever the server pulled, recorded in the database |
| failure mode | a half-finished push leaves the server in a state nobody chose | the server applies a package or does not; it decides when it is ready |
Push wins on immediacy and on making an approval gate legible. Pull wins on the credential direction, which is the one that matters when a repository's CI is compromised, and on servers converging by themselves rather than by being correctly targeted. The immediacy gap closes with a control endpoint; the credential gap does not close at all.
A run producing identical data reloads nothing. A presentation edit renaming a legend regenerates only the manifest.
| Change | Reload data | Regenerate serving | Regenerate manifest |
|---|---|---|---|
| new source data | ✔ | ✔ | |
| a corrected column explanation | ✔ | ✔ | |
| a label, palette or legend | ✔ | ||
| a new layer over an existing column | ✔ | ✔ | |
| a popup field, a zoom range | ✔ | ✔ | |
| a new or changed column | ✔ | ✔ | ✔ |
| regrouping the sidebar | ✔ |
ridescore deploy --dry-run prints these rows for the change in hand, before
anything is applied.
| Order | Why | |
|---|---|---|
| adding | data, then website | the website ignores what it does not know about |
| removing | website, then data | by the time it disappears, nothing asks for it |
| renaming | two separate releases | a rename is an addition followed by a removal; done as one change, it breaks the map |
Proposal 0003 removes most of these cases, since a new layer over existing columns stops being a two-repository change. What still crosses: a new kind of drawing, and any rename.
This is the only place the two release processes interact. How the website itself is released is a separate proposal.
| Development | Production | |
|---|---|---|
| build data | any contributor | never — production only loads |
| pulls | any current package | only promoted packages |
| promote a package | — | a maintainer, deliberately |
Promotion is a retag in the registry, from a manually triggered workflow, never an action taken on a server. Packages and bundles are promoted independently, since a bundle covers any data build of its shape and description revision — a refresh promotes a package alone, and a legend fix promotes a bundle alone. Rolling back is tagging an earlier package; the server takes it on its next tick.
Dev and production apply the same package — nothing is rebuilt between them, which is what makes examining it on dev worth doing. Promotion stays a human decision: it is where somebody takes responsibility for the numbers the public sees.
Reads the load record from each environment and reports what each is serving: which package, loaded when, how many rows, and whether dev and production hold the same one. It asks the database rather than the deployer, so it still answers when the deploying machinery is down or was bypassed.
A person, for now. A build machine checks the sources on a timer and reports what moved — which inputs changed, which commits touched the calculation — and a maintainer reads that and asks for a run. Removing the person is a later and separate decision.
The pipeline runs on a machine that is not serving the public, with capped processor, memory and disk, so a long run slows only itself. Production never builds; it only loads.
GitHub Actions builds and signs, using Environments for the dev/production
distinction and a required reviewer on promotion. An orchestrator would add a
server to operate and would own only the schedule, the fan-out and the approval
gate; ridescoredc/docs/steps/evaluation-kestra.md has the evaluation, which also
notes that an approval gate is more awkward under a pull model than under a push
one — a flag a server polls, rather than a task that pauses. Revisit when two
producers share the street network, when backfills become routine, or when someone
without a terminal needs to trigger a run.
Verification runs after a server applies: the map draws, a slider changes colours, feedback submits, and the version on display is the one that was promoted. The last check is the one that catches a server serving something other than what was promoted, which is otherwise silent.
Source snapshots are kept. Comparing a street against its own past self means re-running old inputs through today's model, so a cache that overwrote itself would destroy the comparison before anyone asked for one. Proposal 0001 §4.2 leaves the policy here.
Published packages get stable, immutable addresses, which is what makes an earlier package redeployable, two runs comparable, and a dataset citable. Rolling back becomes re-pointing at a package rather than rebuilding one.
| Role | Today | Under this proposal |
|---|---|---|
| maintainer promoting | imports data onto a server | retags a package; each server takes it on its next tick |
| security | CI holds an SSH key into production | nothing outside a server can write to it |
| anyone debugging | infers what is live from deploy logs | ridescore status |
| CI | cannot check a presentation without a database | generate runs with no credentials |
| whoever rolls back | re-imports an output somebody kept | applies the previous package |
| Option | Cost | Why the proposal chooses otherwise |
|---|---|---|
| one phase — apply directly | nothing can be examined first, and CI needs a database | generate, then apply |
| serving SQL and the manifest inside the data package | a package would stop being usable by anyone who does not want our map, and a legend fix would republish data | a separate deployment bundle naming the package |
| generate on each server at apply time | a newer image can produce different SQL from the one examined, which defeats promotion | a bundle built once and applied unchanged |
| a bundle pinned to one package digest | every routine refresh would regenerate serving SQL and a manifest that did not change | a bundle declares the shape and description it applies to |
a separate plan command |
a second implementation of generate, drifting from it | the diff falls out of comparing generated artifacts |
| simultaneous data and website releases | cannot be guaranteed; something is always behind | ordered releases |
| push over SSH from CI, as today | a system outside the server holds a key into it, and a server that was down misses the release | servers pull on a timer |
| adopt an orchestrator now | a server to operate, for one pipeline | Actions Environments; revisit under the conditions in §4.7 |
| treat an orchestrator's UI as the record of what is deployed | it reports what it did, not what is true | the database holds the load record |
| Piece | Unblocks | |
|---|---|---|
| E1 |
ridescore generate — package + presentation → a deployment bundle |
CI checking a presentation with no database |
| E2 | ridescore deploy --dry-run |
seeing what a change would do |
| E3 | apply: load the package, apply the bundle's serving SQL, substitute addresses, publish the manifest | a browser never told about a layer before its endpoint exists |
| E4 | ridescore status |
what each environment is serving |
| E5 | packages and bundles published to a registry, signed, addressable by digest | pulling something verifiable |
| E6 | the server-side timer: pull, verify, generate, apply | delivery with no inbound credential |
| E7 | promotion as a retag, from a manually triggered workflow | deliberate promotion, and rollback by retag |
E4 is cheap and useful before any of the rest lands. E3 needs proposal 0005 B1–B3.
| Question | Options | Who decides | |
|---|---|---|---|
| 1 | Which registry, and packages as OCI artifacts or as release assets | the same registry as the image, so one credential and one transport (proposed) / a release asset per package | the team |
| 2 | How long source snapshots are kept | indefinitely — they are small, and are the only way to reproduce an old run (proposed) / a fixed window | the team |
| 3 | Are dev and production reconciled before this lands, or under it? | reconcile first, so a dev deploy is evidence about production / land this and reconcile after | the team. They currently differ in stack and structure |
| 4 | How often a server polls | often enough that a promotion takes effect within an hour (proposed) / on demand only, via the control endpoint | the team |
Comment on the wiki page or in the PR that adds this file.
Feedback that would change the design: §4.1 (the generate/apply split), §4.2 (deciding by content), §4.4 (promotion as a human decision), and §8 question 3, which is about the state of the servers rather than about the design.
Implementation detail: command names, artifact format, which environment generate runs in.