Repository navigation
Proposal 0001
| Status | Draft — for team review |
| Author | Fabian Kloosterman |
| Date | 2026-08-06 |
| Repository | ridescoredc-models |
| Proposes | how data is computed, packaged, contributed to, and loaded into a database |
| Depends on | |
| Related |
0002 adds data descriptions; 0005 and 0006 extend load and deployment. PR #7 (merged), PR #8 (open) |
A proposal is put forward for capturing the generation of Ride Score DC data in an installable and versioned Python library with Command-Line Interface (CLI). The library includes an ordered set of compute stages that (1) fetch data from public sources, (2) construct a base bike road network, and (3) compute safety scores according to one or more models. Stages generate datasets, which are bundled with a record of how they were produced into a versioned, immutable data package. The library includes functions and CLI commands to facilitate executing the stages, generating a data package, loading a data package into a PostGIS database, and converting exploratory notebooks to a new compute stage.
The pipeline exists today as a notebook plus a hand-run ogr2ogr load of
manually saved GeoJSON.
PR #7 (merged) set up the library scaffold. PR #8 ports the notebook's LTS score to a stage as a worked example; it is not proposed for merge as-is. If this proposal is adopted, the notebook's author should create the stage using the library's scaffolding tools.
- Notebooks are good for exploration, not for production. They are not easily testable, reusable, configurable or meaningfully version-controlled. External tooling mitigates some of this, at a cost that makes notebooks less attractive for the exploration they are good at.
- Provenance of computed scores is missing. There is no mechanism to record which sources were used or when they were fetched. This means the computation is not easily reproducible or citable.
-
The database's structure is a by-product of loading GeoJSON.
ogr2ogrinfers column types, the key and the geometry column from a file that carries no schema.
ridescore is an installable, versioned Python library with a CLI that downloads
public source data, computes datasets (e.g., the base bike road network and safety scores; see Appendix A.1), assembles datasets into a versioned data package (see Appendix A.2), and loads a data package into a database. Appendix A.3 shows the library's layout, and Appendix A.4 lists the CLI commands. Each compute stage (e.g., a model; see Appendix A.5) is an ordinary Python function that declares what data it reads and writes, so that the library can build and validate the pipeline (see Appendix A.6).
To ensure data quality and allow the score computations to be reproduced, a data package will contain a record of which data sources were used to compute datasets, the ridescore library version, and the specific configuration (see Appendix A.7) used for the computations. The pipeline proposed in 4.1 already keeps track of which data sources were used to compute datasets.
A data package is one run's datasets bundled with the provenance record
(Appendix A.1, A.2). A data package can be shared with the public ("published") and is used to transfer the scoring pipeline output to the PostGIS database for the Ride Score DC website. A published data package is versioned and never modified (a correcttion is a new package). The version (shape.values) tracks changes to the shape and values of the datasets (Appendix A.2). A data package's files (parquet format) carry their own column types, which facilitates loading the data into PostGIS without implicit conversion (Appendix A.8).
Notebooks are good for quick exploration of data and for designing and evaluation new safety scoring models. However, notebooks are less suited for production (see Section 3 above). Rather than running notebooks in the pipeline, we propose to make it easy to move from a notebook implementation to a production implementation (Appendix A.9). For this, ridescore will include functions to load datasets, data packages, and scoring models, and include a CLI command to scaffold compute stages.
Pros
- The provenance of the data is recorded alongside the data.
- The scoring code becomes ordinary code: reviewable, testable, and reusable. A model can be imported and called elsewhere instead of copied.
- Anyone can easily run the full pipeline with one command. It works on public data, so a new contributor installs the library and runs the pipeline locally without a server or a database.
- The output is a shareable data package.
- Compatible with notebook-driven development of new scoring models.
Cons
- We build and maintain the pipeline execution machinery ourselves.
- You have to think about the code differently. A notebook lets you organise the work however suits you; a module means more organized code that favors testing and reuse.
Prefect, Dagster and Kestra could each take over the pipeline execution part of the library that runs the steps (ordering steps, caching, retry after failure), and each adds a web page showing what ran. The costs are much the same across all three: none of them knows what a dataset, a data package or a version is, so we would still need to implement provenance, bundling, versioning and loading; each is a large dependency with its own opinions to work inside; and the parts most worth having (the web page, the schedule, the history) need a server and a database running, including whenever we build a package for publication. There is also a risk of fighting a framework rather than using it: where its built-in way of doing something not quite fits, we would try to work around it, which is the worst of both worlds.
The three orchestration frameworks differ mainly in fit. Dagster is the closest, because its central idea is also a named dataset produced by one function, and it checks the whole graph before running. Kestra is the furthest, because it would describe the pipeline a second time in a separate file, leaving two descriptions to keep in step. Adopting one of the frameworks at some time in the future is possible, since a stage is a plain decorated function either way (as proposed here).
Phase 0 — scaffold. Done, PR #7. Package layout, CI, lint.
Phase 1 — stages and packages. PR #8, open. fetch | build | run | inspect,
the snapshot, the stage decorator and graph, the package writer. Excludes
descriptions and datapackage.json, which are 0002's.
Phase 2 — the notebook boundary. scaffold for all three kinds, a starting
notebook from road_segment.parquet, a golden test generated from the author's
own output. Depends on Phase 1 only.
Phase 3 — the loader. ridescore load <package> — psycopg COPY,
package-only. Replaces the hand-run ogr2ogr.
Phase 4 — publication. Where packages live and how a version is assigned. Out of scope here; 0006 makes delivery pull-based from a registry.
Breaking changes
-
ogc_fidandwkb_geometrydisappear —ogr2ogrartifacts, not data. Anything keyed toogc_fidbreaks, though feedback keyed to it already detaches on every import, which is the defect this removes. -
The deployed table loses
route_id,slow_streetandlen, which are the notebook's frame dumped wholesale. Confirm nothing downstream reads them. -
Geometry changes from
LineStringZ. Consumers assuming 3D break. -
The map visibly changes —
lts_levelmoves on 2,420 segments andridescore_v1on 2,789. Gated on §8's stress-rule questions; it should not ship as a side effect of adopting the pipeline. -
live/bna/andlive/lts/go. They assert a top-level-per-model layout that conflicts with models being importable subpackages. - The notebook stops being the source of truth for any score that becomes a stage.
| Role | Today | Under this proposal |
|---|---|---|
| model author | edits a shared notebook; someone runs it by hand | adds a stage in one reviewed PR |
| pipeline operator | runs notebook cells, then ogr2ogr
|
ridescore run, then ridescore load <package>
|
| data consumer outside the project | no artifact to use | a versioned data package |
| reviewer | reviews a notebook diff | reviews Python |
- Should the data package conform to the Data Package standard, rather than only resemble it? Weighed together with the Table Schema question in 0002 §5.3, since the two specs are separable.
A dataset is one table's worth of rows: a key, optionally a geometry, and columns. Scores are kept separate from the geometry they describe, so several models can score the same streets while the shapes are stored once.
A geometry dataset's key must stay stable when its source is republished, or anything attached to a feature — another dataset, a user's feedback — comes loose on the next rebuild. Whether the source offers such a key is an open question (§8).
<name>-<version>/
datapackage.json what the data is: identity, version, licences
<dataset>.parquet one file per dataset
run.json how it was produced: which sources, fetched when,
the run date, the library version, the configuration,
and any value a stage derived for itself
The layout resembles the Data Package standard; whether to conform to it is an open question (§8). A published package is never modified — a correction is a new package.
A version is assigned when a package is published, not when one is built, so iterating locally consumes no numbers. At publication the candidate is compared against the last published package:
| Part | Moves when |
|---|---|
| shape | a column is added, removed, renamed or retyped, or a key or geometry changes |
| values | the numbers change and the shape does not |
If nothing differs, nothing is published.
Assembling a package is its own step, so correcting a licence or other metadata recomputes nothing.
src/ridescore/
cli.py the command-line interface
config.py defaults, sectioned by owner (A.7)
load.py a package into PostGIS
sources/ one per public source
crash_records/
roadway_blocks/
geography/ one per shared geometry other stages attach to
road_segment/
census_block/
models/ one per model, each scoring that geometry
ridescore_v1/
tests/
pyproject.toml dependencies, entry point, version
uv.lock the resolved environment
Every part is importable by a predictable name — ridescore.models.ridescore_v1,
and likewise for a source or a geography. Dependencies are pinned, so a
contributor needs the repository and one command to install it.
| Command | |
|---|---|
ridescore fetch |
download each source into a dated snapshot |
ridescore build |
run the compute stages over a snapshot, then bundle |
ridescore bundle |
assemble a data package from what a build produced |
ridescore run |
fetch then build
|
ridescore inspect |
what a run produced, and what moved against another run |
ridescore scaffold <kind> <name> |
start a new stage — source, geography or model
|
ridescore load |
load a data package into a PostGIS database |
ridescore build runs every stage; ridescore build <dataset> runs only the
stage that writes it and the stages it depends on. Each command wraps a Python
function of the same name (§4), so a notebook or a script can do the same.
A stage is one step of the pipeline, and there are two kinds.
A fetch stage downloads one public source through run.get, which writes into
a directory the library provides and records the address, the time and a checksum.
This is the only network access the library offers.
A compute stage declares the datasets it reads and writes:
@stage(
name="ridescore_v1",
reads=["road_segment", "crashes"],
writes=["ridescore_v1_scores"],
)
def build(road_segment, crashes, run):
...
return {"ridescore_v1_scores": scores}Those declarations are what let the library work out the running order and check it (A.6).
Datasets arrive ready to use; sources arrive as files. A dataset was written by the pipeline in a format it controls, so the library hands over a table. A source is whatever the publisher shipped — a zipped shapefile, a spreadsheet with a preamble — so the stage receives the downloaded files and reads them itself. A stage may declare a source alongside its datasets; sources do not affect the running order, since everything is downloaded before any computing starts.
Where a stage lives is organisational. sources/, geography/ and models/
split stages by subject and owner — the pipeline, whoever maintains shared
geometry, and each model's author.
Mistakes in the declarations — two stages writing the same dataset, a dataset read that nothing writes, a circular dependency — are caught before any stage runs, rather than after the street network has been rebuilt.
A stage whose inputs and configuration have not changed is skipped and its previous result reused, so an author tuning a score does not rebuild the street network each time. A build intended for publication always recomputes everything.
Three kinds of setting, separated by whether they can change a published number:
| lives in | recorded | |
|---|---|---|
| how a value is computed — tolerances, windows, thresholds |
config.py, sectioned by owner: pipeline, geography, model |
yes |
| what a model is — the blend's weights |
weights.toml beside the model, checked to sum to 1 |
yes |
| where a run reads and writes — paths, the database address | the invocation and the environment | no |
A change to either of the first two re-runs the stages that depend on it, and
run.json records the values actually used rather than a pointer to a file. The
database address is a credential and is never recorded.
Any setting can be overridden for a single run, and the override is recorded beside the value it replaced. A publication build refuses overrides.
ridescore load <package> takes a package and an empty PostGIS database and
creates a table per dataset, plus a record of which package is held. Column types
and the geometry come from the package's own files, so nothing is guessed from
values and two loads of one package cannot differ. It needs the package and
nothing else, so anyone can do it with their own database.
The key and the indexes need 0002. The package's files carry types but not a primary key, so on this proposal alone the loader creates a table without one.
Scoring data is full-replace. One package is loaded at a time, replacing what was there.
To develop a new scoring method, read a package the pipeline has already built and iterate in Jupyter — the geometry is assembled, so the work starts at scoring.
To turn it into a stage, save what the notebook produced, run
ridescore scaffold model <name>, move the scoring function into it, build, and
use ridescore inspect to confirm the answer has not moved. Then commit and open
a pull request.
To work on shared geometry or a new source, the same loop, starting from a snapshot rather than a package. Comparing matters most there: every model reads the street network, so rebuilding it changes every score downstream.