Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 12 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,17 +135,24 @@ views-postprocessing/
│ ├── source_metadata.py # producer (datafactory) facts
│ ├── store_metadata.py # prediction-store facts
│ └── launch_config.py # the delivery mode the launcher must declare
├── unfao/ # WHO A DELIVERY IS FOR — the only FAO-specific code
├── unfao/ # WHO A DELIVERY IS FOR — the FAO-specific code
│ ├── product.py # targets, consumer name, S_MIN, upload interlock
│ ├── appwrite_env.py # the declared store coordinates
│ └── managers/unfao.py # UNFAOPostProcessorManager
├── crafd/ # WHO A DELIVERY IS FOR — the CRAF'd-specific code
│ ├── product.py # same three files, same shape (register C-33 on
│ ├── appwrite_env.py # why the manager is a copy, and what would
│ └── managers/crafd.py # make it time to stop copying)
└── data/gaul_lookup.parquet # the precomputed GAUL lookup (ADR-011)
```

**Dependencies point one way only:** `unfao/` → `contract/` → `delivery/`. Nothing in
`contract/` may import `unfao/` — that is what lets a new partner reuse the machinery
without inheriting FAO, and it is enforced by `tests/test_clone_readiness.py`, not by
convention. See [`docs/CLONING.md`](docs/CLONING.md).
**Dependencies point one way only:** `<partner>/` → `contract/` → `delivery/`. Nothing
in `contract/` may import a partner package — that is what lets a new partner reuse the
machinery without inheriting another partner's product, and it is enforced by
`tests/test_clone_readiness.py`, not by convention. The partner list lives in one place
(`tests/conftest.py`) and is itself checked against the filesystem, so a package added
without being declared fails rather than passing quietly.
See [`docs/CLONING.md`](docs/CLONING.md).

---

Expand Down
30 changes: 25 additions & 5 deletions docs/ADRs/012_revised_ontology.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,21 @@ packages answer three different questions, and every category below names the on
delivery/ what makes a delivery VALID — representation-free invariants
contract/ how a delivery is BUILT — partner-neutral machinery
unfao/ who a delivery is FOR — one partner's product and manager
crafd/ who a delivery is FOR — another partner's product and manager
```

**Amended 2026-08-03 (#211): the third row repeats.** `crafd/` joined `unfao/` as a
second partner package. This does not add a fourth category — a partner package is a
partner package, and the closed set is unchanged. What it changes is that *"who a
delivery is FOR"* is answered N times rather than once, and every claim below that said
**"the"** partner or **"one"** module now says how many.

A partner package is the shape a partner takes **inside this repository**. It is not
the same thing as a partner *repository*: `views-crafdapi` and `views-productionapi`
are consumer APIs cut from `views-faoapi`, and this repository is the single producer
that serves all of them. `docs/CLONING.md` was written before that was settled and
described the cut-a-repo case; it now says which is which.

| Category | Purpose | Authority | Stability |
|----------|---------|-----------|-----------|
| **Delivery Invariants** | Representation-free rules over primitives that a delivery must satisfy: coverage, no-collapse, gid parity, observed-range, provenance. Live in `delivery/` — **nothing there imports pandas or views_frames**. *Forecast identity was one of these until 2026-07-31 — see the amendment below.* | Authoritative — they define what a valid delivery is | Stable — changes are governance decisions |
Expand All @@ -51,8 +64,8 @@ unfao/ who a delivery is FOR — one partner's product and manager
| **Artifact Builders** | `contract/historical.py` — turns a frame plus the lookup into the partner-facing artifact. | Derived | Evolving |
| **External Facts** | Facts read from systems this repo does not own: the producer's (`contract/source_metadata.py` — `last_valid_month_id`, D-07) and the store's (`contract/store_metadata.py`). | Authoritative (the owning system is the source of truth) | Evolving |
| **Launch Declarations** | `contract/launch_config.py` — the delivery mode the launcher must declare. Omitting a key is **refused by name**, never inferred (ADR-003, register C-63). | Authoritative | Stable |
| **Partner Product** | `unfao/product.py` (targets, consumer document name, collapse floor, upload interlock) and `unfao/appwrite_env.py` (the store coordinates). **This is what a clone replaces.** | Authoritative — one reason to change: the partner relationship | Evolving |
| **Pipeline Manager** | `unfao/managers/unfao.py` — a concrete pipeline-core postprocessor (Template-Method subclass) that *orchestrates* read/transform/validate/save and **calls** the invariants, never inherits them. **It is the only module in the repository that imports `views_pipeline_core`** — the coupling C-40 describes is one file wide. It is not yet *thin*: 406 lines, down from 636 (#149). | Derived | Evolving |
| **Partner Product** | Per partner: `<partner>/product.py` (targets, consumer document name, collapse floor, upload interlock) and `<partner>/appwrite_env.py` (the store coordinates). Two exist — `unfao/` and `crafd/` (#211). **This pair plus the Pipeline Manager below is what a new partner supplies** — three files, as `docs/CLONING.md` states them. | Authoritative — one reason to change: that partner relationship | Evolving |
| **Pipeline Manager** | One per partner: `<partner>/managers/<partner>.py` — a concrete pipeline-core postprocessor (Template-Method subclass) that *orchestrates* read/transform/validate/save and **calls** the invariants, never inherits them. **These are the only modules in the repository that import `views_pipeline_core`**, pinned to an explicit allowlist by `tests/test_doc_accuracy.py` — the coupling C-40 describes is one file per partner. Neither is yet *thin* — each sits just under the 450-line budget `tests/test_doc_accuracy.py` holds them to, down from 636 (#149), and since #211 the second is a near-verbatim copy of the first — deliberate WET with a named extraction trigger, recorded in register **C-33**. | Derived | Evolving |
| **Derived Outputs** | Arrow shards, the GAUL sidecar, the run manifest and the historical parquet, produced per run and delivered to the partner store. | Ephemeral | Ephemeral |

**Two claims this ADR made until 2026-08-01, both now corrected rather than quietly dropped**
Expand All @@ -78,9 +91,16 @@ waiting for an audit.

- **Screaming architecture:** the categories match the package layout — `delivery/`
(invariants), `contract/` (the machinery: `wire/`, the `frame_extraction.py` seam, the
GAUL asset, artifact builders, external-fact readers), `unfao/` (the partner's product and
its manager). A reader can infer responsibilities from the structure, and the one-way
dependency `unfao/ → contract/ → delivery/` is enforced by test, not convention.
GAUL asset, artifact builders, external-fact readers), and one package per partner —
`unfao/`, `crafd/` — each holding that partner's product and its manager. A reader can
infer responsibilities from the structure, and the one-way dependency
`<partner>/ → contract/ → delivery/` is enforced by test, not convention — **both legs
of it, for every declared partner, since #211.** Neither was fully true before:
`test_contract_package_does_not_import_any_partner` named `unfao`, so `contract/` was
free to import `crafd`; and the `contract/ → delivery/` leg had no test at all until
`test_the_invariants_do_not_import_the_machinery` was written — a documented arrow
that half existed, which is worse than an undocumented one because a reader stops
checking.
- **DIP / OCP:** primitives are the abstraction the invariants depend on; the representation
seam is the single point of change for a representation migration (C-40), so the invariants
are closed against it.
Expand Down
2 changes: 1 addition & 1 deletion docs/CICs/UNFAOPostProcessorManager.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ The following **must never** fail silently:
- PRIO-GRID geometry details
- The internals of how the lookup table was built

This anchors the class within ADR-002 (topology): `unfao/` → `contract/` → `delivery/`, one way only. It is **the repository's only importer of `views_pipeline_core`** (mechanically pinned by `tests/test_doc_accuracy.py`), which is what makes C-40's blast radius one file wide. 406 lines as of epic #148, down from 636 — not yet *thin*, and held under a 450-line budget by the same test.
This anchors the class within ADR-002 (topology): `unfao/` → `contract/` → `delivery/`, one way only — and since #211 the same holds for `crafd/`, the second partner package. It is one of **the repository's only two importers of `views_pipeline_core`** (both mechanically pinned to an allowlist by `tests/test_doc_accuracy.py`), which keeps C-40's blast radius at one file per partner. Not yet *thin*: it came down from 636 lines at epic #148 and now sits just under a **450-line budget**, which `tests/test_doc_accuracy.py` applies to the whole `managers/` directory of each partner rather than to this file alone — a seam that holds its line count by moving 800 lines into a sibling module has not held anything. The exact figure is deliberately not repeated here; the test carries it.

---

Expand Down
105 changes: 89 additions & 16 deletions docs/CLONING.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,19 @@
# Cloning this repository for a new partner

Read this **before** cutting `views-crafdapi`, `views-productionapi`, or any future
partner delivery. It is short on purpose.
# Adding a new partner delivery

Read this **before** adding a partner package to this repository, or cutting a partner
API repository that consumes one. It is short on purpose.

> **Corrected 2026-08-03 (#211). This document used to say "before cutting
> `views-crafdapi`", and that framed the job wrongly.** `views-crafdapi` and
> `views-productionapi` are **consumer** APIs, cut from `views-faoapi`. They do not
> clone *this* repository. This repository is the single **producer** that serves every
> partner, and a new partner is added here as a package alongside `unfao/` and
> `crafd/` — not as a new producer repo.
>
> The three-things-to-supply structure below survives that correction unchanged, because
> it was always describing the same three files. What changes is where they go: into a
> new directory in this repo, not into a new repository. The "Hard rules" section below
> is where the distinction actually mattered, and it is corrected there too.

## What you get for free

Expand All @@ -16,10 +28,18 @@ partner package arrives with them).

## What you must supply

Three things. They are the only FAO-specific files in the repository, so the shape of
your work is: **replace these three, keep everything else.**
Three things. They are the only partner-specific files in the repository, so the shape
of your work is: **copy these three from an existing partner, change them, keep
everything else.** `crafd/` is the worked example. One caveat if you read it as a
template: it did *not* copy `unfao/managers/README.md`, the operational summary that
sits beside the FAO manager. That was an omission rather than a decision — write one.

Register them as you add them: `tests/conftest.py` holds the repository's single
declared partner list (`PARTNER_PACKAGES`), and a partner missing from it is exempt
from every guard below. A test asserts that list against the filesystem, so forgetting
fails CI rather than passing quietly — which is what happened when `crafd/` landed.

### 1. Your product — `unfao/product.py`
### 1. Your product — `<partner>/product.py`

Four declarations, and nothing may be inferred:

Expand All @@ -32,25 +52,78 @@ Four declarations, and nothing may be inferred:
- **`UPLOAD_ENABLED`** — the interlock. **Leave it `False`** until your consumer's
selection guard is deployed in production, not merely merged.

### 2. Your store coordinates — `unfao/appwrite_env.py`
### 2. Your store coordinates — `<partner>/appwrite_env.py`

The env-var names your delivery requires, validated fail-loud before any store is
constructed. Names come from the **Appwrite Seam Contract's coordinate registry** (homed in
views-appwrite) and are referenced **by URL at a pinned commit, never copied**. The
secret stays an operator slot.

### 3. Your manager — `unfao/managers/unfao.py`
### 3. Your manager — `<partner>/managers/<partner>.py`

The pipeline-core seam. **The partner managers are the only modules in the repository
that import `views_pipeline_core`**, and a test holds them to an explicit allowlist.
Yours will orchestrate read → transform → validate → save and *call* the invariants —
never inherit them.

Today the two managers are near-identical. See for yourself rather than trusting a
number here — the number went stale twice while this paragraph was being written:

```
diff views_postprocessing/unfao/managers/unfao.py \
views_postprocessing/crafd/managers/crafd.py
```

Sixteen lines differ on each side and **none of them changes behaviour**: the import,
the class name, the two partner-named methods and their two call sites, the four
env-var literals, one line that both selects which env tuple is validated and labels
the store, one runtime refusal message, and four lines of prose.

The pipeline-core seam. **This is the only module in the repository that imports
`views_pipeline_core`**, and a test keeps it that way. Yours will orchestrate
read → transform → validate → save and *call* the invariants — never inherit them.
**That is deliberate** — WET before DRY, and the second copy is what finally showed the
seam is a config object rather than a behavioural one. Register **C-33** carries the
extraction trigger: a **third** in-repo partner, or the first bug that has to be
hand-patched identically in both files. If you are the third, read C-33 before copying
a fourth time.

## Hard rules, and why each exists

**Do not import `views_pipeline_core.modules.{appwrite,datastore}`.**
þing-02 **S24(5)**, binding. This repository's own import of those is how a two-repo
defect became three (register C-40); pipeline-core declines to offer the surface, and
that refusal is deliberate. Write a thin client against the SDK, as views-faoapi did.
**Your manager may import `views_pipeline_core.modules.{appwrite,datastore}`.
Nothing else in this repository may — and you add yourself to the allowlist by hand.**
`tests/test_doc_accuracy.py::test_views_pipeline_core_is_confined_to_the_partner_managers` pins the
importer set to an explicit list of manager files. Adding a partner means editing that
list deliberately. That is the cost of a new partner, not a formality: the coupling is
bounded only because someone has to write the file's name down.

Pipeline-core declines to export that surface, and this repository's own import of it is
how a two-repo defect became three (register **C-40**). Unwinding it is deferred under
issue **#146** behind a **named trigger — þ01-D8's supply trigger firing on the C-221
decomposition, explicitly not on this repository's convenience**. Do not read the
deferral as "not done yet"; it is a decision with a condition attached (ADR-014 §4).

**If you are cutting a consumer API repo, the rule inverts: do *not* import them.**
þing-02 **S24(5)** binds the repositories cut from views-faoapi — `views-crafdapi`
(the þing records call it `un-crafdapi`) and `views-productionapi`. It does not reach a
partner package inside this producer, which is why `crafd/managers/crafd.py` may import
what a consumer API may not. An earlier version of this document cited the verdict as a
flat prohibition and over-claimed it. Write a thin client against the SDK, as
views-faoapi did.

**Check the store's result. It is load-bearing, not boilerplate.**
`_ContractStorePort.upload` inspects `result.success` and raises. It looks like
defensive noise and is not. When metadata storage fails after the file is already
uploaded, the pipeline-core store logs the error and **returns
`OperationResult(success=False, code="PARTIAL_SUCCESS")`** — it reports the failure
faithfully and simply does not raise. A caller that discards the result therefore
proceeds as though the delivery were complete, leaving a file with no metadata
document: invisible to the consumer, exactly like a wrong document name. That happened
to run-0's historical artifact on 2026-07-27.

This matters more than it reads, because of a date. þing-02 **D10/S30** required this
repository's legacy path to be guarded or retired **before 2026-11-30**, when the
current key expires — an unguarded path on that day *reports success and ships nothing*.
The legacy path itself was retired in #149 (register C-63), so what the obligation now
amounts to is keeping this guard on the contract path. If you copy a manager you inherit
it; do not tidy it away.

**Get your own key before the first run, not after.**
Free at t=0, a migration later. One key per identity per environment (the Appwrite Seam Contract
Expand Down
17 changes: 13 additions & 4 deletions docs/architecture/role_and_seams.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,10 @@ metadata, guards their integrity, and delivers them to a partner store** — it
**post-forecast delivery layer**, not a spatial-mapping library and not a statistical
post-processor.

The only live consumer today is the **UN FAO** delivery (`views_postprocessing/unfao/`).
Two partner deliveries live here: the **UN FAO** one
(`views_postprocessing/unfao/`), delivering to FAO-FSFC since 2026-07-27, and
**CRAF'd** (`views_postprocessing/crafd/`), added 2026-08-03 with its upload interlock
still closed. They are peers — one producer, one partner package each.

---

Expand Down Expand Up @@ -182,11 +185,17 @@ views_postprocessing/
│ ├── source_metadata.py producer (datafactory) facts, e.g. last_valid_month_id
│ ├── store_metadata.py prediction-store facts
│ └── launch_config.py the delivery mode the launcher must declare
├── unfao/ WHO A DELIVERY IS FOR — the only FAO-specific code
├── unfao/ WHO A DELIVERY IS FOR — the FAO-specific code, and only that
│ ├── product.py targets, consumer document name, S_MIN, upload interlock
│ ├── appwrite_env.py the declared store coordinates
│ └── managers/unfao.py UNFAOPostProcessorManager (406 lines; the only importer
│ of views_pipeline_core)
│ └── managers/unfao.py UNFAOPostProcessorManager
├── crafd/ WHO A DELIVERY IS FOR — the CRAF'd-specific code (same three
│ │ files, same shape; register C-33 on why it is a copy)
│ ├── product.py
│ ├── appwrite_env.py
│ └── managers/crafd.py CRAFDPostProcessorManager
│ the two managers are the ONLY importers of
│ views_pipeline_core — one per partner, allowlisted by test
└── data/gaul_lookup.parquet the precomputed GAUL lookup (ADR-011)
```

Expand Down
Loading
Loading