Skip to content

Year Built Prediction Pipeline

Jump to bottom
Jakub Ziolkowski edited this page Oct 14, 2026 · 7 revisions

Year Built Prediction Pipeline

The Year Built Prediction Pipeline is an automated, machine learning and computer vision pipeline that estimates construction years for buildings lacking construction records in Poland's official topographic database (BDOT10k).

The pipeline pairs a historical aerial orthophoto time-series detector (DiGi.YOLO) with detector-only dating — the first-confident-detection rule (DiGi.GIS.ML) — fully orchestrated and database-backed via DiGi.GIS.YOLO.UI.ConsoleApp.

Tip

To actually run it, start at Running a County. That page carries the standing operator recipe — which runs fill which features, the order they must go in, the readiness check, and how to verify what was stored. This page explains how the pipeline works; that one says what to do.

1. Problem Statement & Conceptual Design

Poland's central building database (BDOT10k) records construction years for only a portion of the building stock. The pipeline determines missing construction years by analyzing historical aerial orthophoto imagery captured between 2008 and the present day:

Historical Orthophotos (Geoportal / orto_datas)
                       │
                       ▼
         [Step 1: Export Image Crops]
                       │
                       ▼
       [Step 2: YOLO Detection (DiGi.YOLO)]
       Scores building presence per year
                       │
                       ▼
       [Step 3: Parse Detection Series]
       Confidence vectors per building (2008..now)
                       │
                       ▼
     [Step 4: Update building_data (Optional)]
     Persists 90 detection columns to PostgreSQL
                       │
                       ▼
     [Step 5: Dating (DiGi.GIS.ML)]
     First confident detection ≥ 0.5, fallback to first detection
                       │
                       ▼
     [Step 6: Persist Predictions (Optional)]
     Updates year_built_data & building_data

The Two-Stage Serial Architecture

  1. YOLO Object Detector (DiGi.YOLO): A frozen object detection checkpoint trained exclusively on the Building category. For each building, the detector evaluates historical aerial crops across every available year to determine visibility and bounding box confidence.
  2. Dating Rule (DiGi.GIS.ML): Dates each building from its per-year detection series alone — the first year the detector saw it at confidence ≥ 0.5, falling back to the first year it was seen at all — and stamps first-confident-detection@0.5 on the stored prediction. The ML.NET regressor that used to occupy this step is parked as the evaluation baseline (DiGi.GIS.ML#15).

Note

A building the detector never fires on is not dated, and a Predicted year built an earlier run stored for it is cleared rather than left stale (DiGi.GIS.YOLO.UI#28). The user entry is never touched, and a building with no stored imagery was never exported, so it is neither dated nor cleared.

Presence-vs-Absence Training

The detector is trained using buildings whose construction years are definitively known:

  • Aerial orthophoto crops taken in or after the known construction year carry positive bounding box annotations.
  • Aerial orthophoto crops taken before the construction year are supplied with empty label files (negative examples).

This enables the detector to learn visual presence versus absence across historical landscape changes, making its per-year detection confidence curve a direct temporal indicator of construction.

2. Model Weights & Provenance

The production pipeline relies on a single, frozen checkpoint located in the git-ignored user files/YOLO/models/model.pt (copied to bin/YOLO/models/model.pt during compilation). Since 2026-10-08 that is train9_fresh, selected by the detector gate of DiGi.GIS.YOLO.UI#12 — SHA-256 2dd833e438ffcb1352ae7675063c9f0c97ca1c39ce6410a0bfab96e8b3cbdc9a. Dating is detector-only: every stored prediction carries first-confident-detection@0.5 instead of a model SHA-256, and the regressor retrained on this checkpoint's detections (DiGi.GIS.ML#13) is kept as the evaluation baseline rather than run in production. The full provenance, read from the checkpoint, is in the README.

Property train9_fresh (shipped) train8 (previous; the rollback)
Architecture YOLO26x YOLOv8x
Saved 2026-10-07 2025-05-23
Framework Version ultralytics 8.4.165 ultralytics 8.3.130
Epochs 150 (best 145) 150 (on top of train7)
Image size / batch / seed / optimizer 640 / 16 / 0 / auto 640 / 16 / 0 / auto
Started from base/yolo26x.pt (COCO pretrained) runs/detect/train7/weights/best.pt
Dataset 25 008 labelled buildings rebuilt from the database lost
SHA-256 2dd833e438ffcb1352ae7675063c9f0c97ca1c39ce6410a0bfab96e8b3cbdc9a c79d2edfb776c913494d698cd1899ed1a6c92d7caad594b674d39ce5ada7ea38

Note

Weights are frozen and not tracked in source control. Only model.pt is deployment content; earlier and candidate weights are kept in a training directory outside the workspace. There is no paired model to keep in sync — dating reads the detection columns this checkpoint wrote, so a county must be rewritten with them before it is dated (DiGi.GIS.YOLO.UI#23).

3. Environment & Prerequisites

Python Dependencies

Prediction execution requires Python 3.9+ with dependencies installed from requirements.txt:

pip install -r requirements.txt

The dependencies are pinned for reproducibility:

  • ultralytics==8.4.165: Pinned exactly, because a different ultralytics can be a different detector. The shipped YOLO26x checkpoint was trained and written by 8.4.165 and does not load under 8.3.130; the runner's preflight refuses an older version. 8.4.165 was adopted after it reproduced train8's detections exactly (#17).
  • torch>=2.6: CUDA-enabled PyTorch build recommended. Inference throughput reaches 75–90 images/second on GPU versus ~5 images/second on CPU.

Environment Preflight Check

The pipeline includes an automated environment preflight check (DiGi.YOLO.Query.YOLOEnvironmentResult). Before downloading imagery or allocating resources, the orchestrator verifies:

  1. Python interpreter availability.
  2. Successful import of ultralytics and torch.
  3. Model weights existence and readability.

If the environment preflight fails, the process immediately exits with exit code 2, preventing wasted data downloads.

4. Pipeline Orchestration & Execution

Execution of the complete pipeline is handled unattended by the headless console runner: DiGi.GIS.YOLO.UI.ConsoleApp.exe (from DiGi.GIS.YOLO.UI).

There are two ways to start it, and both end in that same executable:

  • By hand, with an options file — every flag available, which is what the samples below describe.
  • From the tray application (DiGi.GIS.PostgreSQL.UI → Predict year built), which asks only for the run's scope and then runs the full six-step flow: the task forces the six step and write flags on (ExportImages, RunPrediction, Score, UpdateDetections, UpdateYearBuiltData, UpdatePredictedYearBuilt), fixes Resume: true, and leaves Years, Radiuses, Confidence and ModelPath at their defaults — CleanScratchDirectory is the one flag it offers as a choice. On a deployed host the runner is an optional extension of that application, at <DiGi.GIS.PostgreSQL.UI>\extensions\DiGi.GIS.YOLO.UI.ConsoleApp, so a machine without the ML payload simply does not offer the task. The standing operator recipe is Running a County.

Options Configuration (YearBuiltPredictionPipelineOptions.json)

{
  "CountyIds": [104106],
  "ScratchDirectory": "scratch",
  "ModelPath": "user files/YOLO/models/model.pt",
  "Confidence": 0.1,
  "MaxConcurrentRequests": 8,
  "BatchSize": 5000,
  "ReferenceBatchSize": 10000,
  "Resume": true,
  "ExportImages": true,
  "RunPrediction": true,
  "Score": true,
  "UpdateDetections": false,
  "UpdateYearBuiltData": false,
  "UpdatePredictedYearBuilt": false
}

Scoping to a County (Database Identifiers vs Codes)

Important

Use County Database IDs, Never Territorial Codes: BDOT10k stores county administrative boundaries in administrative_areal_2d by polygon parts. A territorial code (such as "2212" for powiat słupski) is not a database primary key.

Counties whose territory comprises multiple detached polygons have multiple database rows (e.g. county 2212 corresponds to database row IDs 73482 and 73485). Supplying a territorial code will match 0 rows. The pipeline's Query.UnknownCountyIds validator verifies provided IDs up front and fails fast if an invalid code is supplied.

Step Breakdown & Safe Dry Runs

All database write steps (UpdateDetections, UpdateYearBuiltData, UpdatePredictedYearBuilt) default to false. This guarantees that initial passes over a new county are read-only dry runs:

  1. Imagery is downloaded and saved to <ScratchDirectory>/<CountyId>/.
  2. YOLO inference creates <ScratchDirectory>/<CountyId>/results.bbrf.
  3. Inferred construction years are dated (first confident detection, fallback to first detection) and summarized to the console.
  4. Enabling the write flags in a subsequent pass performs bulk updates to PostgreSQL via api.digiproject.uk.

5. Superseding the Legacy File-Based Workflow

This automated pipeline completely supersedes and replaces the legacy six-step manual process previously documented in Year build predictions.docx:

Historical Manual Process vs Automated Replacement

Stage Legacy Manual Workflow (Year build predictions.docx) Automated Pipeline Replacement
1. Export User manually clicked UI button Convert_ToFiles in DiGi.GIS.UI to unpack .odf files into folders. Handled automatically via WebAPI stream retrieval (Modify.ExportPredictionImagesAsync).
2. Inference User manually opened terminal, cd'd into script directory, and ran python predict.py. Orchestrated unattended with preflight validation and crash resumption.
3. Detection Grouping User manually clicked UI button Modify.AppendBuilding2DYearBuiltPredictionsFile to emit .bypf. Parsed in-memory and translated into structured data representations.
4. Feature Table User manually clicked UI button Modify.AppendPredictionTable to generate a .tsv table. Built dynamically with paging through api.digiproject.uk.
5. Regressor Scoring User manually edited hard-coded paths in OrtoBuildingDetectionModelConsoleApp Program.cs and executed it. Invoked programmatically via DiGi.GIS.ML.Classes.YearBuiltPredictor — detector-only dating.
6. Output Writing User clicked Modify.AppendPredictionYearBuilts to write .ybdf files, then uploaded them via tray task. Bulk-persisted directly to PostgreSQL via authenticated WebAPI endpoints.

Retention of Permanent File Formats

Per DiGi.YOLO#10, deprecating the manual workflow does not mean deleting the file formats:

  • All core file format classes (.gmf, .odf, .sdcf, .bbrf, .bypf, .ybdf) remain active, permanent capabilities for offline data export, archival snapshots, and interoperability across the GIS toolchain.
  • Only the manual file-only UI buttons and modification entry points in DiGi.GIS.UI have been deprecated with [Obsolete] and TODO [YearBuiltPipelineFileSource] tags per sub-task S16.

6. Related Documentation & Guides

Clone this wiki locally