Repository navigation
Proposal 0002
| Status | Draft — for team review |
| Author | Fabian Kloosterman |
| Date | 2026-08-07 |
| Repository | ridescoredc-models |
| Proposes | how a dataset is described (metadata) |
| Depends on | 0001 — stages, datasets and data packages |
| Related | 0003 resolves the manifest against these descriptions; 0004 adds recipes to the grammar; 0005 and 0006 build and deploy tables from them |
Every dataset a stage writes gets a dataset description: a committed file stating what each column means, its unit, where it came from, and how rows are keyed. A command fills in what a machine can know and the author writes the prose; every build checks the data against its description and fails on disagreement; and every data package carries a copy, which is what lets a package explain itself to someone who will never see this repository.
Proposal 0001 produces datasets and bundles them into packages recording how they were produced. Nothing in it records what the data means. This adds that.
A dataset does not carry enough to interpret it. The files record column
names, types and values. They do not say what a column measures, in what units,
where it came from, or how to read a missing value. A package adds how the data
was produced, which is not the same thing: a reader can see which sources were
fetched and when, and still not know what parking_presence records.
The website cannot say anything about the data it draws unless somebody hardcodes it there. A label, a unit on a legend, a sentence in a popup — each written into the website by hand, because there is nowhere else to read it from. The same fact ends up written twice, the two drift, and every new column needs a website change before anyone can be told what it is.
A dataset description is a committed file beside the stage that produces the dataset, stating what each column means, its unit, where it came from, how rows are keyed, and how they relate to other datasets (A.1, A.2). It is a claim about data, so it lives with the code that produces the data rather than with the website that draws it, and it says nothing about how anything is drawn.
ridescore describe <dataset> reads a dataset the pipeline has produced and
writes or updates its description: an entry for every new column with the
machine-known fields filled, types and keys updated in place, a column the data no
longer carries marked rather than silently dropped, and never a field a person
wrote overwritten (A.3). The author supplies the rest — what the value means, its
unit, where it came from. describe --validate checks that every file is well
formed and has no unfilled marker left; it reads no data, so it runs on every
change.
A build compares what each stage produced against its description and fails on any disagreement — a missing or unexpected column, a mismatched type, a duplicate or null key, the wrong geometry, or a value outside the range the author declared (A.4). A declared range is a claim rather than an observation, which is what makes it worth checking: a range re-derived from each run can never be violated, while a declared one catches the run that produces a score of 140.
A data package carries a copy of every description, which is what lets it explain itself to someone who will never see this repository. Because a description can change while the numbers do not, the package version gains a third part (A.5), and correcting a sentence costs a re-bundle rather than a rebuild. Computations a description might one day declare (e.g., the weights of a blended score) are 0004's subject.
Pros
- A package explains itself. Someone who was never involved can read what each column measures and in what units.
- The website reads labels and units from one place. Today the popup field list exists three times over and the copies disagree.
- A bad run is caught. A range the author declared catches a score of 140; an unexplained column cannot reach a published package.
- Descriptions are checked without running the pipeline, so a mistake is found on the change that introduced it.
Cons
- A second thing to keep current for every dataset. The build check is what stops it drifting, but it is still more to write.
- An unexplained column blocks publication. Correct, and it will be felt as friction the first time someone adds one.
- A new column cannot be described until its stage has run once, because the command reads produced data.
- Another grammar of our own to maintain, with its own reader and its own versioning.
This is the current state: a column is explained by a comment, a markdown cell, or nobody, and the website hardcodes its own labels. It costs nothing to keep, and anyone can write a sentence wherever they like. But the explanation never reaches anyone who does not have that notebook, the same fact written in two places drifts apart, and every new column needs a website change before it can be labelled.
Frictionless Table Schema is the closest fit, and 0001 already models the data
package on its parent, the Data Package standard. Its primaryKey, foreignKeys,
constraints.minimum/maximum, constraints.enum and constraints.required
cover almost everything §4 asks for, and frictionless-py validates data against
a schema, which is most of the checking work off the shelf. Two gaps: it has no
field for a unit, and its geospatial support does not declare a geometry type or a
coordinate system — though GeoParquet already carries both in the file itself.
Extending Table Schema is a designed-for case rather than a workaround. Tools must
ignore properties they do not recognise, so a unit or a geometry declaration can
simply be added; and a profile — a JSON Schema the package declares
conformance to — can make our additions required and checkable, which is how
Camtrap DP and Fiscal Data Package extend the same standard. Proposal
0004's recipe would be a custom field property on the same
footing, and the grammar version A.2 records becomes the profile the package
declares. One cost: the standard validator ignores what it does not know, so our
own additions still need our own checking.
dbt's schema.yml covers column descriptions and value checks in a committed
file, and could travel in a package. But its vocabulary assumes SQL models
materialised in a warehouse, it has no geometry concept, and adopting the file
format without the tool means writing our own reader — which is most of what
rolling our own costs.
A catalog service — DataHub, OpenMetadata, Amundsen — describes tables in a running system. That metadata cannot travel inside a package to somebody who has neither our database nor our tooling, and it cannot be checked in CI without the service running.
All phases depend on 0001's Phase 1.
D0 — the grammar and describe --validate. A reader and the check. Needs no
data, so it lands first and independently.
D1 — ridescore describe. Seeding and merging from produced data.
D2 — the check inside build.
D3 — descriptions carried in the package, and the third version part.
Breaking changes
-
The package version gains a part. Anything parsing
shape.valuesbreaks. Cheap now, because nothing outside this repository consumes a package yet. buildstarts failing on datasets it previously wrote.
Each amends 0001 rather than replacing anything in it:
| In 0001 | Amendment |
|---|---|
| 0001 A.1, datasets | one dataset, one producing stage, one description |
| 0001 A.2, the data package | carries a copy of every description, and the grammar version |
| 0001 A.2, the version | gains a third part — see A.5 below |
| 0001 A.4, commands | gains ridescore describe and describe --validate
|
| 0001 A.5, stages | the check of §4.3 is applied to whatever a stage returns |
| 0001 A.6, re-running | a description does not decide a re-run, so an edited description is re-checked without recomputing anything |
| 0001 A.8, loading | a table gains its key, its indexes and its declared nullability from the description. Column types already come from the package's files |
| 0001 A.9, contributing | gains a describe step between the first build and the second |
- Should declared relationships be checked?
- What happens to a marked-removed column?
- Should we adopt Table Schema, Data Package, both, or neither? (§5.3)
dataset: road_segment
title: The DC road network, as segments
description: >
One row per roadway block, carrying the geometry and the street
attributes that models score.
key: [segment_id]
geometry:
column: geometry
type: LineString
crs: EPSG:4326
columns:
- name: segment_id
type: string
nullable: false
description: Identifier of the roadway block.
source: DDOT roadway blocks, field BLOCKKEY
- name: speed_limit
type: int16
unit: mph
description: >
Posted speed limit. Filled from the road class where the
source does not state one.
source: DDOT roadway blocks, field SPEEDLIMITOne file per dataset, beside the module that produces it. Nothing in it depends on data values: it states shape and meaning, so none of it moves when a source is refetched and the numbers change. Seeding is not an exception — the command uses data to suggest a range, but what is committed is the author's claim.
Dataset level
| Field | ||
|---|---|---|
dataset |
machine | the dataset's name |
title |
author | a short human name |
description |
author | what one row is, and what the dataset is for |
key |
machine | the columns that identify a row |
geometry |
machine |
column, type and crs; absent when there is no geometry |
references |
author | how this dataset attaches to another |
columns |
both | one entry per column, in the order the data carries them |
Column level
| Field | ||
|---|---|---|
name, type, nullable
|
machine | type comes from the vocabulary below |
unit |
author | a physical unit, where the number has one |
description |
author | what the value means |
source |
author | a source and its field, or the stage that computed it |
values |
both | for a category column: each permitted value and what it means |
range |
both | for a numeric column: what the column is allowed to hold |
Types. A small vocabulary mapping onto both parquet and PostgreSQL without
negotiation: string, bool, int16, int32, int64, float32, float64,
date, timestamp, geometry.
Markers. Any author field the command created and nobody has written holds
TODO. A TODO anywhere fails --validate, but not build, because a
description is allowed to be incomplete while a computation is still being worked
out.
Grammar version. The package records the grammar as a single integer. A consumer refuses a version above the one it knows rather than guessing. Description files in the repository carry no version: they ship with the library that reads them.
ridescore describe <dataset> reads a dataset the pipeline has already produced.
Given one with no description, it creates it. Run again after the data changes, it
merges rather than overwrites:
-
adds an entry for each new column, machine-known fields filled and every
prose field set to
TODO; - updates machine-known fields in place — type, nullability, key, geometry;
- marks a column the data no longer carries, rather than deleting it silently;
- never rewrites a field a person wrote.
It then reports what it changed and what needs attention.
describe --build runs the producing stage first if its output is not current, so
the loop can be entered from the other end. It is opt-in; the plain command never
runs a pipeline, and errors if there is no output to read.
The check runs on every build and covers a missing or unexpected column, a mismatched type, a duplicate or null key, the wrong geometry type or projection, and a value outside a declared range.
It gates the package, not the stage. A stage that adds a column disagrees with the description that existed before it, and the command needs the new data to record that column. That looks circular and is not: the check runs after a stage's output has been written, so a failing build still leaves output to describe. What it withholds is the package.
road_segment: data and description disagree
+ parking_setback int16 (in the data, not in the description)
- slow_street (in the description, not in the data)
run: ridescore describe road_segment
Build order is data first, description second. An author gets the computation right with no description at all, then generates one and writes what a machine cannot determine.
0001 versions a package shape.values. A description changes on its own, so it
gets a third:
| Part | Moves when | Resets |
|---|---|---|
| shape | a column is added, removed, renamed or retyped | — |
| values | the numbers change and the shape does not | on a shape change |
| description | prose, a unit or a declared range changes | on a shape change |
Values and description are independent: a refresh moves the middle part and leaves the last alone, and correcting a sentence does the reverse. A shape change resets both.