# Year Built Prediction Pipeline

The **Year Built Prediction Pipeline** is an automated, machine learning and computer vision pipeline that estimates construction years for buildings lacking construction records in Poland's official topographic database (BDOT10k).

The pipeline pairs a historical aerial orthophoto time-series detector (**DiGi.YOLO**) with detector-only dating — the first-confident-detection rule (**DiGi.GIS.ML**) — fully orchestrated and database-backed via **DiGi.GIS.YOLO.UI.ConsoleApp**.

> [!TIP]
> **To actually run it, start at [Running a County](https://github.com/DiGi-Organization/DiGi.GIS.YOLO.UI/wiki/Running-a-County).** That page carries the standing operator recipe — which runs fill which features, the order they must go in, the readiness check, and how to verify what was stored. This page explains how the pipeline works; that one says what to do.

---

## 1. Problem Statement & Conceptual Design

Poland's central building database (BDOT10k) records construction years for only a portion of the building stock. The pipeline determines missing construction years by analyzing **historical aerial orthophoto imagery** captured between 2008 and the present day:

```
Historical Orthophotos (Geoportal / orto_datas)
                       │
                       ▼
         [Step 1: Export Image Crops]
                       │
                       ▼
       [Step 2: YOLO Detection (DiGi.YOLO)]
       Scores building presence per year
                       │
                       ▼
       [Step 3: Parse Detection Series]
       Confidence vectors per building (2008..now)
                       │
                       ▼
     [Step 4: Update building_data (Optional)]
     Persists 90 detection columns to PostgreSQL
                       │
                       ▼
     [Step 5: Dating (DiGi.GIS.ML)]
     First confident detection ≥ 0.5, fallback to first detection
                       │
                       ▼
     [Step 6: Persist Predictions (Optional)]
     Updates year_built_data & building_data
```

### The Two-Stage Serial Architecture

1. **YOLO Object Detector (`DiGi.YOLO`):** A frozen object detection checkpoint trained exclusively on the `Building` category. For each building, the detector evaluates historical aerial crops across every available year to determine visibility and bounding box confidence.
2. **Dating Rule (`DiGi.GIS.ML`):** Dates each building from its per-year detection series alone — the first year the detector saw it at confidence ≥ 0.5, falling back to the first year it was seen at all — and stamps `first-confident-detection@0.5` on the stored prediction. The ML.NET regressor that used to occupy this step is parked as the evaluation baseline ([DiGi.GIS.ML#15](https://github.com/DiGi-Organization/DiGi.GIS.ML/issues/15)).

> [!NOTE]
> A building the detector never fires on is not dated, and a `Predicted year built` an earlier run stored for it is **cleared** rather than left stale ([DiGi.GIS.YOLO.UI#28](https://github.com/DiGi-Organization/DiGi.GIS.YOLO.UI/issues/28)). The user entry is never touched, and a building with no stored imagery was never exported, so it is neither dated nor cleared.

### Presence-vs-Absence Training

The detector is trained using buildings whose construction years are definitively known:
- Aerial orthophoto crops taken **in or after** the known construction year carry positive bounding box annotations.
- Aerial orthophoto crops taken **before** the construction year are supplied with **empty label files** (negative examples).

This enables the detector to learn visual presence versus absence across historical landscape changes, making its per-year detection confidence curve a direct temporal indicator of construction.

---

## 2. Model Weights & Provenance

The production pipeline relies on a single, frozen checkpoint located in the git-ignored `user files/YOLO/models/model.pt` (copied to `bin/YOLO/models/model.pt` during compilation). Since **2026-10-08** that is `train9_fresh`, selected by the detector gate of [DiGi.GIS.YOLO.UI#12](https://github.com/DiGi-Organization/DiGi.GIS.YOLO.UI/issues/12) — SHA-256 `2dd833e438ffcb1352ae7675063c9f0c97ca1c39ce6410a0bfab96e8b3cbdc9a`. Dating is detector-only: every stored prediction carries `first-confident-detection@0.5` instead of a model SHA-256, and the regressor retrained on this checkpoint's detections ([DiGi.GIS.ML#13](https://github.com/DiGi-Organization/DiGi.GIS.ML/issues/13)) is kept as the evaluation baseline rather than run in production. The full provenance, read from the checkpoint, is in the [README](https://github.com/DiGi-Organization/DiGi.YOLO#frozen-weights-provenance).

| Property | `train9_fresh` (shipped) | `train8` (previous; the rollback) |
|---|---|---|
| **Architecture** | YOLO26x | YOLOv8x |
| **Saved** | 2026-10-07 | 2025-05-23 |
| **Framework Version** | `ultralytics` 8.4.165 | `ultralytics` 8.3.130 |
| **Epochs** | 150 (best 145) | 150 (on top of `train7`) |
| **Image size / batch / seed / optimizer** | 640 / 16 / 0 / `auto` | 640 / 16 / 0 / `auto` |
| **Started from** | `base/yolo26x.pt` (COCO pretrained) | `runs/detect/train7/weights/best.pt` |
| **Dataset** | 25 008 labelled buildings rebuilt from the database | lost |
| **SHA-256** | `2dd833e438ffcb1352ae7675063c9f0c97ca1c39ce6410a0bfab96e8b3cbdc9a` | `c79d2edfb776c913494d698cd1899ed1a6c92d7caad594b674d39ce5ada7ea38` |

> [!NOTE]
> Weights are frozen and not tracked in source control. Only `model.pt` is deployment content; earlier and candidate weights are kept in a training directory outside the workspace. There is no paired model to keep in sync — dating reads the detection columns this checkpoint wrote, so a county must be rewritten with them before it is dated ([DiGi.GIS.YOLO.UI#23](https://github.com/DiGi-Organization/DiGi.GIS.YOLO.UI/issues/23)).

---

## 3. Environment & Prerequisites

### Python Dependencies

Prediction execution requires Python 3.9+ with dependencies installed from `requirements.txt`:

```bash
pip install -r requirements.txt
```

The dependencies are pinned for reproducibility:
- **`ultralytics==8.4.165`**: Pinned exactly, because a different ultralytics can be a different detector. The shipped YOLO26x checkpoint was trained and written by 8.4.165 and does not load under 8.3.130; the runner's preflight refuses an older version. 8.4.165 was adopted after it reproduced `train8`'s detections exactly ([#17](https://github.com/DiGi-Organization/DiGi.YOLO/issues/17)).
- **`torch>=2.6`**: CUDA-enabled PyTorch build recommended. Inference throughput reaches **75–90 images/second on GPU** versus ~5 images/second on CPU.

### Environment Preflight Check

The pipeline includes an automated environment preflight check (`DiGi.YOLO.Query.YOLOEnvironmentResult`). Before downloading imagery or allocating resources, the orchestrator verifies:
1. Python interpreter availability.
2. Successful import of `ultralytics` and `torch`.
3. Model weights existence and readability.

If the environment preflight fails, the process immediately exits with exit code `2`, preventing wasted data downloads.

---

## 4. Pipeline Orchestration & Execution

Execution of the complete pipeline is handled unattended by the headless console runner:
**`DiGi.GIS.YOLO.UI.ConsoleApp.exe`** (from [DiGi.GIS.YOLO.UI](https://github.com/DiGi-Organization/DiGi.GIS.YOLO.UI)).

There are **two ways to start it**, and both end in that same executable:

* **By hand**, with an options file — every flag available, which is what the samples below describe.
* **From the tray application** (`DiGi.GIS.PostgreSQL.UI` → *Predict year built*), which asks only for the run's scope and then runs the full six-step flow: the task **forces the six step and write flags on** (`ExportImages`, `RunPrediction`, `Score`, `UpdateDetections`, `UpdateYearBuiltData`, `UpdatePredictedYearBuilt`), fixes `Resume: true`, and leaves `Years`, `Radiuses`, `Confidence` and `ModelPath` at their defaults — `CleanScratchDirectory` is the one flag it offers as a choice. On a deployed host the runner is an optional extension of that application, at `<DiGi.GIS.PostgreSQL.UI>\extensions\DiGi.GIS.YOLO.UI.ConsoleApp`, so a machine without the ML payload simply does not offer the task. The standing operator recipe is [Running a County](https://github.com/DiGi-Organization/DiGi.GIS.YOLO.UI/wiki/Running-a-County).

### Options Configuration (`YearBuiltPredictionPipelineOptions.json`)

```json
{
  "CountyIds": [104106],
  "ScratchDirectory": "scratch",
  "ModelPath": "user files/YOLO/models/model.pt",
  "Confidence": 0.1,
  "MaxConcurrentRequests": 8,
  "BatchSize": 5000,
  "ReferenceBatchSize": 10000,
  "Resume": true,
  "ExportImages": true,
  "RunPrediction": true,
  "Score": true,
  "UpdateDetections": false,
  "UpdateYearBuiltData": false,
  "UpdatePredictedYearBuilt": false
}
```

### Scoping to a County (Database Identifiers vs Codes)

> [!IMPORTANT]
> **Use County Database IDs, Never Territorial Codes:**
> BDOT10k stores county administrative boundaries in `administrative_areal_2d` by polygon parts. A territorial code (such as `"2212"` for powiat słupski) is **not** a database primary key.
> 
> Counties whose territory comprises multiple detached polygons have multiple database rows (e.g. county `2212` corresponds to database row IDs `73482` and `73485`). Supplying a territorial code will match 0 rows. The pipeline's `Query.UnknownCountyIds` validator verifies provided IDs up front and fails fast if an invalid code is supplied.

### Step Breakdown & Safe Dry Runs

All database write steps (`UpdateDetections`, `UpdateYearBuiltData`, `UpdatePredictedYearBuilt`) default to `false`. This guarantees that initial passes over a new county are read-only dry runs:
1. Imagery is downloaded and saved to `<ScratchDirectory>/<CountyId>/`.
2. YOLO inference creates `<ScratchDirectory>/<CountyId>/results.bbrf`.
3. Inferred construction years are dated (first confident detection, fallback to first detection) and summarized to the console.
4. Enabling the write flags in a subsequent pass performs bulk updates to PostgreSQL via `api.digiproject.uk`.

---

## 5. Superseding the Legacy File-Based Workflow

This automated pipeline completely **supersedes and replaces** the legacy six-step manual process previously documented in `Year build predictions.docx`:

### Historical Manual Process vs Automated Replacement

| Stage | Legacy Manual Workflow (`Year build predictions.docx`) | Automated Pipeline Replacement |
|---|---|---|
| **1. Export** | User manually clicked UI button `Convert_ToFiles` in `DiGi.GIS.UI` to unpack `.odf` files into folders. | Handled automatically via WebAPI stream retrieval (`Modify.ExportPredictionImagesAsync`). |
| **2. Inference** | User manually opened terminal, cd'd into script directory, and ran `python predict.py`. | Orchestrated unattended with preflight validation and crash resumption. |
| **3. Detection Grouping** | User manually clicked UI button `Modify.AppendBuilding2DYearBuiltPredictionsFile` to emit `.bypf`. | Parsed in-memory and translated into structured data representations. |
| **4. Feature Table** | User manually clicked UI button `Modify.AppendPredictionTable` to generate a `.tsv` table. | Built dynamically with paging through `api.digiproject.uk`. |
| **5. Regressor Scoring** | User manually edited hard-coded paths in `OrtoBuildingDetectionModelConsoleApp` `Program.cs` and executed it. | Invoked programmatically via `DiGi.GIS.ML.Classes.YearBuiltPredictor` — detector-only dating. |
| **6. Output Writing** | User clicked `Modify.AppendPredictionYearBuilts` to write `.ybdf` files, then uploaded them via tray task. | Bulk-persisted directly to PostgreSQL via authenticated WebAPI endpoints. |

### Retention of Permanent File Formats

Per [DiGi.YOLO#10](https://github.com/DiGi-Organization/DiGi.YOLO/issues/10), deprecating the manual workflow does **not** mean deleting the file formats:
- All core file format classes (`.gmf`, `.odf`, `.sdcf`, `.bbrf`, `.bypf`, `.ybdf`) remain **active, permanent capabilities** for offline data export, archival snapshots, and interoperability across the GIS toolchain.
- Only the manual file-only UI buttons and modification entry points in `DiGi.GIS.UI` have been deprecated with `[Obsolete]` and `TODO [YearBuiltPipelineFileSource]` tags per sub-task S16.

---

## 6. Related Documentation & Guides

* **[Running YOLO Scripts Manually](Running-Scripts-Manually.md):** Guide for running `predict.py`, `train.py`, and `export.py` directly from the command line.
* **[Running a County (DiGi.GIS.YOLO.UI)](https://github.com/DiGi-Organization/DiGi.GIS.YOLO.UI/wiki/Running-a-County):** The standing operator recipe — run order, county readiness check, options files, verification and exit-code triage. **Start here to run the pipeline.**
* **[ConsoleApp Operator Guide (DiGi.GIS.YOLO.UI)](https://github.com/DiGi-Organization/DiGi.GIS.YOLO.UI/wiki/Using-the-ConsoleApp):** Complete operational manual for `DiGi.GIS.YOLO.UI.ConsoleApp.exe` with configuration samples, recovery procedures, and exit codes.
* **[Year Built Prediction (DiGi.GIS.ML)](https://github.com/DiGi-Organization/DiGi.GIS.ML/wiki/Year-Built-Prediction):** The dating rule, the stored-year contract and the parked regressor's performance metrics.
* **[DiGi.YOLO Wiki Home](Home.md):** Repository landing page.