Skip to content

Using the ConsoleApp

Jump to bottom
Jakub Ziolkowski edited this page Oct 14, 2026 · 8 revisions

Using DiGi.GIS.YOLO.UI.ConsoleApp

The DiGi.GIS.YOLO.UI.ConsoleApp is the headless prediction runner and WebAPI-backed pipeline orchestrator for Year Built prediction. It runs unattended, fetching orthophoto imagery from api.digiproject.uk, executing YOLO object detection, reading the stored per-year detection columns, and persisting the predicted construction year back to PostgreSQL.

Tip

This page is the reference for every option, sample and exit code. For the standing order — which runs must fill which features, in what sequence, before this runner can date a county at all — read Running-a-County first.

1. Pipeline Overview & Execution Steps

The pipeline coordinates six modular steps per county:

Step 1: Export Images          -> Downloads building-level orthophoto crops into scratch/
Step 2: Run YOLO Inference     -> Detects building bounding boxes per year into results.bbrf
Step 3: Parse Detections       -> Extracts per-year detection confidences
Step 4: Update Building Detections -> Bulk-writes detection columns to building_data (optional)
Step 5: Detector-only Dating       -> Dates detected buildings by first confident detection
Step 6: Update Predictions         -> Writes to year_built_data and building_data (optional)
Cleanup: Remove scratch/<countyId>  -> Only if the county had no failed step, and CleanScratchDirectory is true

Each step is governed by its own boolean flag in the configuration file. This allows safe, read-only dry runs, selective step execution, and cheap resuming when recovering from network drops or server timeouts.

2. Prerequisites & Environment Setup

System Requirements

Component Requirement Details
Operating System Windows 10 / 11 / Server Re-encoding stored orthophoto bytes relies on System.Drawing.Image, which requires Windows (net10.0-windows7.0).
.NET Runtime .NET 10 Runtime Required to run DiGi.GIS.YOLO.UI.ConsoleApp.exe.
Python Python 3.9+ with ultralytics & torch A CUDA-enabled GPU build is strongly recommended — inference runs at 75–90 images/second on GPU vs ~5 images/second on CPU.
YOLO Weights model.pt Place under user files/YOLO/models/model.pt. Copied to bin/YOLO/models/model.pt during build.
ML.NET Model OrtoBuildingDetectionModel.mlnet Shipped beside the executable but retired from production — the dating rule needs no model file.
Client API Key user files/GIS_WebAPI_Client.conf Contains Key="<secret>". Required for write operations.

Tip

The runner automatically executes an environment preflight check (DiGi.YOLO.Query.YOLOEnvironmentResult) before downloading any imagery. If Python, ultralytics, or PyTorch is missing or misconfigured, the process exits within seconds with exit code 2, avoiding unnecessary image exports.

The weights are preflighted alongside the interpreter, resolved exactly as the county loop resolves them, and a path that resolves to nothing is refused with exit code 2 — the environment code, alongside the interpreter — before the export rather than after it. That check was added because it was missing: YOLOEnvironmentResult probes only a model it was actually given, so a ModelPath of null passed the preflight silently and then failed on the first county — after that county's imagery had already been exported.

Server-Side Requirements (Write Operations)

When writing detections or predicted years, the target WebAPI host enforces tiered API key authorization. In GIS_WebAPI.conf on the server:

Enabled=true
Key="<matching-client-key>"
AllowUpdateBuildingData=true
AllowUpdateYearBuiltData=true

Warning

AllowUpdateBuildingData reads GetValue<bool> with no default, so an absent flag evaluates to false. If omitted on the host, every write attempt returns HTTP 401 Unauthorized regardless of whether a valid key is supplied.

3. Configuration Reference (YearBuiltPredictionPipelineOptions)

The pipeline is configured via a JSON options file (by default YearBuiltPredictionPipelineOptions.json in the application root).

Option Type Default Description
CountyIds List<int> [] Database identifiers (id) of the target counties (never territorial codes). Mandatory.
ScratchDirectory string "scratch" Directory where downloaded images and detection results are stored. Mandatory.
ModelPath string "user files/YOLO/models/model.pt" Path to YOLO model weights (.pt). Resolves relative to bin or user files.
PythonPath string? null Path to python.exe. When null, automatically discovered from system PATH.
WorkingDirectory string? null Execution directory for YOLO scripts. When null, resolves to the application base directory.
Confidence double 0.1 Detection confidence threshold (0.0 to 1.0).
Years Range<int>? null Target orthophoto years, as a range object ({ "Min": 2008, "Max": 2025 }) — not a list. When null, defaults to 2008..2025. The run checks it against the range the predictor states it reads before it reads a county — the first-detection rule states exactly that default — and a range narrower than it is refused (exit code 1), a wider one runs with the surplus years reported as ignored. Leave it null (or match the stated range).
Radiuses List<double>? null Radii of the radial-ratio feature columns. When null, the column list applies its own default. It is checked against the predictor the same way as Years: a set that omits a radius it reads is refused (exit code 1), one that adds a radius runs with the surplus reported. Leave it null for the same reason as Years.
MaxConcurrentRequests int 8 Maximum concurrent image download requests.
BatchSize int 5000 Chunk size for bulk database update queries.
ReferenceBatchSize int 10000 Maximum building references requested per feature query (hard-capped at 10 000).
Resume bool true If true, skips image files already present on disk in the scratch directory.
CleanScratchDirectory bool true Deletes <ScratchDirectory>/<CountyId>/ once the run has finished with that county — only if that county recorded no failed step. A failed county keeps its imagery and detections so a re-run is cheap; a cancelled one is cleaned. Set false for the split detections-then-score workflow, whose second run reads results.bbrf off disk, and for any run you intend to resume.
ExportImages bool true Step 1: Download and extract building orthophoto crops.
RunPrediction bool true Step 2: Run YOLO inference over exported images to create results.bbrf.
Score bool true Step 5: Date buildings from their stored detection columns (first-confident-detection rule).
UpdateDetections bool false Step 4: Write 90 detection columns to building_data.
UpdateYearBuiltData bool false Step 6a: Write timestamped prediction history to year_built_data.
UpdatePredictedYearBuilt bool false Step 6b: Update the latest year in building_data."Predicted year built".

Caution

CleanScratchDirectory and Resume pull in opposite directions — but only after a county that succeeded. With cleanup on, a county that completes leaves nothing on disk, so Resume cannot help a later run pick it up. That is the deliberate price of closing the gap where a cleaned scratch directory silently skipped a county whose detection columns were already stored. A county that failed keeps its folder, so correcting the cause and re-running costs seconds rather than repeating the export (~30 min) and the inference (~85 min on CUDA). Set CleanScratchDirectory: false whenever a later run has to read what a successful run wrote — the split workflow being the case that needs it.

Caution

Spelling & Schema Matching: Deserialization drops unknown or misspelled JSON properties silently. Always refer to YearBuiltPredictionPipelineOptions.json to verify exact casing.

Caution

Years and Radiuses decide which columns the projection asks for, and the run checks them against the predictor before it reads a county. The predictor states the range and radiuses it reads — the first-detection rule states exactly the options' defaults (2008..2025, the default radiuses) — and the options are compared against that. A projection that omits a feature the predictor reads is refused with exit code 1, naming the columns it would otherwise have defaulted to zero; one that adds features it does not read is reported as a warning and the surplus is ignored. Leave both null unless you are deliberately matching a predictor's stated contract. DiGi.GIS.ML#6.

4. Step-by-Step Execution Guide

Step 1: Navigate to the Output Directory

Always execute from the directory holding the executable and its resolved assets. Where that is depends on how the machine got the runner:

Location
Workspace checkout DiGi.GIS.YOLO.UI\bin
Deployed host <DiGi.GIS.PostgreSQL.UI>\extensions\DiGi.GIS.YOLO.UI.ConsoleApp

On a deployed host the runner ships as an optional extension of the tray application rather than as a folder of its own — a machine flagged as ML-capable receives it, and one that will never score a building does not. The weights, OrtoBuildingDetectionModel.mlnet and the runner's own GIS_WebAPI_Client.conf sit beside the executable either way.

# workspace checkout
Set-Location "DiGi.GIS.YOLO.UI\bin"

# deployed host
Set-Location "<DiGi.GIS.PostgreSQL.UI>\extensions\DiGi.GIS.YOLO.UI.ConsoleApp"

Important

Running from outside that directory causes relative paths (such as ScratchDirectory or ModelPath) to resolve against unexpected working directories.

Warning

The YearBuiltPredictionPipelineOptions.json sitting beside the executable may not be the committed default. CopyUserFiles overwrites it with whatever the build machine had in user files, and that copy travels to the deployed host. Check its CountyIds and its write flags before running it by name.

Step 2: Configure Client Authorization

Ensure your WebAPI authorization key is present in GIS_WebAPI_Client.conf beside the runner's own executable:

Key="your-authorization-key"

Important

There are two of these files and they are unrelated. The tray application authorizes with the one beside it; the runner authorizes with the one beside itself, even when the tray application started the run. An exit code 3 from a tray-started run is almost always the runner's copy.

Step 3: Write the Detections, Then Dry-Run

Caution

A read-only dry run is not the first pass. Dating reads the 90 detection columns from building_data that this pipeline writes itself. On a county whose detections have never been written, a run with Score: true is refused with exit code 4, naming the missing detection group and the run that fills it — that is the guard working, not a misconfiguration.

So the first pass over a new county writes the detections (UpdateDetections: true, Score: false), and the dry run comes second. The full order, including the Update building data runs that fill the other 82 features, is in Running-a-County.

Once the detections are stored, run with the remaining write flags disabled to verify feature coverage and dating without touching year_built_data or the Predicted year built column:

.\DiGi.GIS.YOLO.UI.ConsoleApp.exe dry_run_options.json

Inspect the summary printed on completion:

=================================================
 Pipeline execution completed successfully!
 Buildings: 5412
 Images: 48910
 Detections: 48802
 Predictions: 5410
 Building Data updated: 0
 Year Built Data updated: 0
=================================================

Step 4: Execute Full Production Pipeline

Enable the write flags (UpdateDetections, UpdateYearBuiltData, UpdatePredictedYearBuilt) in your configuration and execute:

.\DiGi.GIS.YOLO.UI.ConsoleApp.exe production_options.json

5. Practical Configuration Samples

Sample 0: Detections Only (First Pass Over a New County)

Exports the imagery, runs the detector and writes the 90 detection columns, without scoring anything. This is what makes the county scorable; run it before Sample 1 or Sample 3:

{
  "CountyIds": [104106],
  "ScratchDirectory": "scratch",
  "ModelPath": "user files/YOLO/models/model.pt",
  "Confidence": 0.1,
  "MaxConcurrentRequests": 8,
  "BatchSize": 5000,
  "Resume": true,
  "CleanScratchDirectory": false,
  "ExportImages": true,
  "RunPrediction": true,
  "Score": false,
  "UpdateDetections": true,
  "UpdateYearBuiltData": false,
  "UpdatePredictedYearBuilt": false
}

Sample 1: Safe Dry-Run (Read-Only Evaluation)

Downloads orthophotos, executes YOLO inference, and computes predicted construction years without updating the database. Valid only on a county whose detection columns are already stored — see the caution in §4 Step 3; on a fresh county run Sample 0 first.

{
  "CountyIds": [104106],
  "ScratchDirectory": "scratch",
  "ModelPath": "user files/YOLO/models/model.pt",
  "Confidence": 0.1,
  "MaxConcurrentRequests": 8,
  "BatchSize": 5000,
  "Resume": true,
  "ExportImages": true,
  "RunPrediction": true,
  "Score": true,
  "UpdateDetections": false,
  "UpdateYearBuiltData": false,
  "UpdatePredictedYearBuilt": false
}

Sample 2: Full End-to-End Production Run

Runs all six stages, updating both building_data and year_built_data:

{
  "CountyIds": [104106],
  "ScratchDirectory": "scratch",
  "ModelPath": "user files/YOLO/models/model.pt",
  "Confidence": 0.1,
  "MaxConcurrentRequests": 8,
  "BatchSize": 5000,
  "ReferenceBatchSize": 10000,
  "Resume": true,
  "ExportImages": true,
  "RunPrediction": true,
  "Score": true,
  "UpdateDetections": true,
  "UpdateYearBuiltData": true,
  "UpdatePredictedYearBuilt": true
}

Sample 3: Fast Resume / Recovery Run

If a run fails during database writes (e.g., due to a temporary network drop or missing server permissions), reuse the already downloaded images and YOLO detections on disk:

{
  "CountyIds": [104106],
  "ScratchDirectory": "scratch",
  "Resume": true,
  "CleanScratchDirectory": false,
  "ExportImages": false,
  "RunPrediction": false,
  "Score": true,
  "UpdateDetections": true,
  "UpdateYearBuiltData": true,
  "UpdatePredictedYearBuilt": true
}

Tip

Setting ExportImages: false and RunPrediction: false skips downloading tens of thousands of images and hours of inference, directly reusing <ScratchDirectory>/<CountyId>/results.bbrf to complete the database writes in seconds.

Important

This only works if the run that produced results.bbrf was configured with CleanScratchDirectory: false. With the default the file is gone, and the run refuses the county with exit code 4 naming the path it looked for — it no longer reports it as a county with no detections.

Sample 4: Multi-Part County Configuration

Poland has 18 counties whose territorial area is split into multiple disconnected polygons. BDOT10k stores one row per polygon part in administrative_areal_2d. For example, county code 2212 (powiat słupski) consists of parts 73482 and 73485.

{
  "CountyIds": [73482, 73485],
  "ScratchDirectory": "scratch",
  "ModelPath": "user files/YOLO/models/model.pt",
  "Confidence": 0.1,
  "ExportImages": true,
  "RunPrediction": true,
  "Score": true,
  "UpdateDetections": true,
  "UpdateYearBuiltData": true,
  "UpdatePredictedYearBuilt": true
}

6. Exit Codes & Diagnostics

DiGi.GIS.YOLO.UI.ConsoleApp.exe communicates execution status via process exit codes:

Exit Code Meaning Remediation
0 Success Pipeline finished all requested stages with no step failures.
1 Configuration / Argument Error Check that the options file exists, CountyIds contains positive integers, and ScratchDirectory is non-empty. Verify that IDs were supplied, not territorial codes. Or the Years/Radiuses projection omits features the predictor reads — the feature-contract refusal, which names the missing columns: widen them to the stated range.
2 Preflight Environment Failure Python environment check failed. Verify that Python 3.9+, ultralytics, and PyTorch are installed in the active environment or specified via PythonPath.
3 Authorization Key Missing GIS_WebAPI_Client.conf could not be found or contains an empty Key value.
4 Pipeline Execution Failure One or more steps encountered unhandled errors or server-side rejections — or the run was refused before scoring. Read the [NOTE] lines first: a county identifier that is in no county row, or the detection group that carries nothing, each name the run that fixes them. Otherwise check console error output and Serilog logs.
5 Cancelled Execution cancelled by the user via Ctrl+C. Written rows stay committed. Cancelling is not a failure, so the county in progress is still cleaned unless CleanScratchDirectory: false — set it before starting a run you expect to interrupt.

7. Key Operational Gotchas

  1. County IDs vs Territorial Codes: Never use territorial codes (e.g. "2212", "1465"). A county code is not a primary key. Query.UnknownCountyIds validates every ID against stored county rows before running. If a code is detected, it terminates the run immediately and logs the exact database identifiers required.
  2. Only Detected Buildings are Dated: If the YOLO detector does not detect a building across any orthophoto year, that building has no detection series — the dating rule's only input. It is intentionally skipped rather than dated on an invalid blank vector, and a Predicted year built an earlier run stored for it is cleared rather than left stale (#28). Its user entry is untouched, and a building with no stored imagery was never exported, so it is neither dated nor cleared.
  3. Orthophoto Year Coverage is Variable: Different counties have different available imagery years in Geoportal. Years with no coverage return zero images and are safely bypassed.
  4. Graceful Cancellation (Ctrl+C): Pressing Ctrl+C triggers an orderly cancellation. Already written rows remain committed in PostgreSQL. Scratch files do not remain intact by default — cancelling is a deliberate act rather than a failure, so the county in progress is still cleaned and a restart repeats the export and the inference. Set CleanScratchDirectory: false before starting a run you expect to interrupt. (A county that fails is treated differently: it keeps its folder.)
  5. A Missing results.bbrf Is A Failure, Not An Empty County: With RunPrediction: false the run rebuilds its building list from <ScratchDirectory>/<CountyId>/results.bbrf, never from the detection columns already in building_data. If that file is absent the county is reported as a failed step with exit code 4, naming the path. It used to be logged as a county with no detections and skipped in silence, which is indistinguishable from a county the detector genuinely found nothing in — and only one of the two is recoverable.

8. Related Documentation

Clone this wiki locally