Skip to content

Training the Detector

Jump to bottom
Jakub Ziolkowski edited this page Oct 14, 2026 · 4 revisions

Training the Detector

How to train the YOLO building detector that Predict year built scores imagery with, in two ways: a fresh model from the pretrained yolo26x.pt, or by continuing training from an existing model such as the deployed model.pt. The page covers what each needs, complete option files you can copy, how a run is supervised and recovered, and what to keep so that the model can be trained again later.

The procedure was established by #12 (train9, October 2026). The numbers on this page were measured there.

1. What a training run is

The detector answers one question per orthophoto: is this building already standing in this year's photo? A building's estimated year built is the first year it is detected, its first detection year. Training teaches that from buildings whose year built a user entered.

building_data."User year built"  ┐
building_2d (footprints)         ├─ api.digiproject.uk ─► dataset builder ─► <Dataset>\images, labels, conf.yaml, manifest
orto_datas (orthophotos by year) ┘                                                │
                                                                                  ▼
             start weights (yolo26x.pt or an existing model) ─► ultralytics training ─► <Run>\<RunName>.pt
                                                                                  │
                                         Validation on Test + Detector evaluation ◄┘  (the gate table)

One run is DiGi.GIS.YOLO.UI.ConsoleApp --train <options file>, which runs up to five steps in a fixed order and stops at the first failure:

# Step (Steps value) What it does Typical time (train9)
1 Dataset Builds the dataset from the database, or appends new buildings to an existing one. Resumable. hours: one orthophoto request per labelled building
2 LabelCheck Runs the current detector over 500 Train/Validate images and reports how well the label boxes match (IoU). minutes
3 Train ultralytics training from StartWeightsPath. 30–37 min per epoch, 2–4 days per run
4 Validate mAP of the new weights on the Test split. 6 min
5 Evaluate First-detection-year MAE / RMSE / exact share of every gate weights file and the new weights, on the full Test set and its clean subset. 10–15 min

You can start it two ways:

  • The tray task Train YOLO detector in DiGi.GIS.PostgreSQL.UI (recommended for real runs). It fills in and checks the options, supervises the run, and offers to resume an interrupted one. See the tray wiki page YOLO detector training.
  • By hand, with an options file (this page's samples). Every option is available, for diagnostics and recovery.

The dataset rules (labels, the positive/negative images, the box, duplicates, the split and the Legacy flag) are in Running-a-County §12 and the repository README, section The YOLO training dataset.

2. Before any run

Requirement Detail
GPU machine train9 ran on an RTX 5090 (32 GB). It used 15–17 GB of VRAM for YOLOv8x and 19–21 GB for YOLO26x at batch 16, image size 640, with AMP on. Free the GPU first: no LM Studio model, other training, or GPU-gated DiGi.Test facts during the run.
Python CPython 3.13 with ultralytics 8.4.165 and torch 2.7.0+cu128 (the dataset folder's requirements.txt records the pin). YOLO26 models do not load under ultralytics 8.3.x.
Runner DiGi.GIS.YOLO.UI.ConsoleApp with --train, deployed at <DiGi.GIS.PostgreSQL.UI>\extensions\DiGi.GIS.YOLO.UI.ConsoleApp (written <Runner> below), with GIS_WebAPI_Client.conf and Data_2025.05.27.tsv beside it.
Weights <Runner>\YOLO\models\model.pt (the incumbent) and <Runner>\YOLO\models\base\yolo26x.pt (the fresh base; its SHA-256 is in the DiGi.YOLO README).
Labels current Run the Year Built building-data update for the counties first. The labels are read from the User year built column it writes.
Disk About 3.3 GB for a 228 569-image dataset, plus 0.7–2.3 GB per run folder.
The machine stays up Two to four days of training: turn off sleep, postpone Windows Update restarts, keep the tray running. Closing the tray stops a tray run.

Every path in an options file should be absolute: the runner makes relative paths absolute against its current directory, not against the options file.

3. Train a fresh model (from yolo26x.pt)

Use this to train a model with a fully known history: it starts from the generic pretrained YOLO26x and learns only from this project's dataset. train9_fresh (150 epochs, about 88 h) is the reference run.

3a. Fresh model on a new dataset (build everything)

A fresh dataset is built into a new, empty folder. Resume: false refuses a folder that already holds a dataset, so an existing one is never overwritten.

D:\YOLO\runs\train10_fresh.YOLOTrainingRunOptions.json:

{
  "DatasetOptions": {
    "CountyIds": [5, 204, 310, 314],
    "OutputDirectory": "D:\\YOLO\\train10_dataset",
    "CountOnly": false,
    "Resume": false,
    "LegacyReferencesFilePath": "C:\\ProgramData\\DiGi\\DiGi.GIS.PostgreSQL.UI\\extensions\\DiGi.GIS.YOLO.UI.ConsoleApp\\Data_2025.05.27.tsv",
    "LegacyCutoff": "2025-05-23T00:00:00+00:00",
    "Offset": 1,
    "ValidateWeight": 0.1,
    "HoldoutDenominator": 5,
    "Seed": 0,
    "MaxConcurrentRequests": 8,
    "ReferenceBatchSize": 10000,
    "ModelPath": "C:\\ProgramData\\DiGi\\DiGi.GIS.PostgreSQL.UI\\extensions\\DiGi.GIS.YOLO.UI.ConsoleApp\\YOLO\\models\\model.pt",
    "Confidence": 0.1,
    "LabelCheckSampleSize": 500,
    "LabelCheckOverlayCount": 20,
    "ReportsDirectory": "D:\\YOLO\\reports",
    "WeightsPaths": [
      "C:\\ProgramData\\DiGi\\DiGi.GIS.PostgreSQL.UI\\extensions\\DiGi.GIS.YOLO.UI.ConsoleApp\\YOLO\\models\\model.pt"
    ]
  },
  "StartWeightsPath": "C:\\ProgramData\\DiGi\\DiGi.GIS.PostgreSQL.UI\\extensions\\DiGi.GIS.YOLO.UI.ConsoleApp\\YOLO\\models\\base\\yolo26x.pt",
  "RunName": "train10_fresh",
  "ProjectDirectory": "D:\\YOLO\\runs",
  "Epochs": 150,
  "Patience": 50,
  "ImageSize": 640,
  "Batch": 16,
  "Seed": 0,
  "Device": "0",
  "AutoResumeCount": 10,
  "InactivityTimeout": null,
  "ResumeTraining": false,
  "Steps": ["Dataset", "LabelCheck", "Train", "Validate", "Evaluate"]
}
  • CountyIds are database ids, one per county part: a multi-part county has several. Select every labelled county. The tray's Select All writes all 406 parts, and parts without labels add nothing.
  • Keep HoldoutDenominator 5, Seed 0, ValidateWeight 0.1, Offset 1 and the Legacy* settings unchanged between runs. HoldoutDenominator decides which buildings are Test, by reference. Keeping it fixed keeps every Test building unseen by every model trained so far, so gates stay comparable across generations.
  • WeightsPaths are the incumbents the new model is compared with. The runner adds the new weights itself.
  • AutoResumeCount 10: about one interruption per 9 h was seen on the train9 machine. See §5.
  • Check the dataset before training, in two optional extra runs:
    1. Count only: with "CountOnly": true and "Steps": ["Dataset"], the run counts labelled buildings, the split, duplicates and a size estimate, and writes nothing.
    2. Build and check: with "Steps": ["Dataset", "LabelCheck"], the run stops after the label check. Look at the overlays in ReportsDirectory\label_check before starting the training (train9: mean IoU 0.857, 98.8 % ≥ 0.5).

Run it from the runner's folder:

cd "C:\ProgramData\DiGi\DiGi.GIS.PostgreSQL.UI\extensions\DiGi.GIS.YOLO.UI.ConsoleApp"
.\DiGi.GIS.YOLO.UI.ConsoleApp.exe --train "D:\YOLO\runs\train10_fresh.YOLOTrainingRunOptions.json"

Keep that window open for the whole run, which takes days. The tray task runs the same thing in the background and is the better choice for anything unattended.

3b. Fresh model on an existing dataset

train9_fresh reused the dataset train9_continue had been trained on, so that both candidates learned from the same images. Leave out the Dataset and LabelCheck steps and point OutputDirectory at the existing folder. Only the dataset step reads CountyIds and Resume, so they can be left out and the existing dataset is never touched. The tray's dialog still asks for at least one county.

{
  "DatasetOptions": {
    "OutputDirectory": "D:\\YOLO\\train9_dataset",
    "WeightsPaths": [
      "C:\\ProgramData\\DiGi\\DiGi.GIS.PostgreSQL.UI\\extensions\\DiGi.GIS.YOLO.UI.ConsoleApp\\YOLO\\models\\model.pt",
      "D:\\YOLO\\runs\\train9_continue\\train9_continue.pt"
    ]
  },
  "StartWeightsPath": "C:\\ProgramData\\DiGi\\DiGi.GIS.PostgreSQL.UI\\extensions\\DiGi.GIS.YOLO.UI.ConsoleApp\\YOLO\\models\\base\\yolo26x.pt",
  "RunName": "train9_fresh",
  "ProjectDirectory": "D:\\YOLO\\runs",
  "Epochs": 150,
  "Patience": 50,
  "Device": "0",
  "AutoResumeCount": 10,
  "Steps": ["Train", "Validate", "Evaluate"]
}

Members left out take their defaults (the README's Options tables).

3c. What to expect from a fresh run

  • Val mAP50-95 starts low and climbs for most of the run: train9_fresh went from 0.747 (epoch 1) to 0.868 (16), 0.905 (50), 0.925 (100) and 0.9327 (best, epoch 145).
  • A small dip just after epoch 3 is the end of the learning-rate warm-up, and is normal.
  • The last 10 epochs run without mosaic augmentation (close_mosaic 10) and usually add a final 0.001–0.003.
  • Early stopping (Patience 50) rarely ends a run early: the linear learning-rate decay keeps producing new highs. Budget for the full ceiling.

4. Continue training an existing model

"Continuing" means a new run that starts from an existing model's weights: the deployed model.pt, or a staged candidate such as model_train9_fresh.pt. It keeps everything the model learned and adds what the new data teaches. train9_continue (from train8, 100 epochs, about 50 h) is the reference run.

Continuing is not the same as resuming:

Continue (this section) Resume (§5)
Starts from a finished model (<RunName>.pt, model.pt) an interrupted run's weights\last.pt
Run a new run name and folder, epoch 1 of a new schedule the same run, at the epoch where it stopped
Hyperparameters yours (Epochs, Patience, …) restored from the checkpoint; cannot change
Optimizer state none, starts fresh restored

A finished model cannot be resumed: its checkpoint is marked finished (epoch -1) and carries no optimizer state.

4a. The data you need to continue training

Keep these after every training run. Without the irreplaceable ones, a later run cannot be compared fairly, or not run at all.

Irreplaceable: archive them off the training machine

Item Where (train9) Why it is needed
The model weights, <RunName>.pt, with their SHA-256 <Project>\<RunName>\ (train9_fresh: 118 MB, 2dd833e4…) The start point. The file carries the architecture, the weights and train_args (the hyperparameters and data path it was trained with).
The dataset manifest dataset_references.tsv <Dataset>\ (train9: 1.5 MB, 25 007 buildings) The record of what the model has seen: reference, county, split (Train / Validate / Test), label, Legacy. An append uses it to skip buildings already built; a later gate uses it to prove its test buildings are unseen.
The run record: args.yaml, results.csv, the run's *.YOLOTrainingRunOptions.json files <Project>\<RunName>\, <Project>\ The exact settings, county ids and learning curve, for provenance (DiGi.YOLO README) and for comparing the next run.
The environment pin <Dataset>\requirements.txt ultralytics 8.4.165 / torch 2.7.0+cu128 / Python 3.13. A model is safest trained and scored under the version that wrote it.

Rebuildable but expensive: keep them too

Item Size (train9) Rebuild cost
Dataset images and labels (<Dataset>\images, labels, conf.yaml) 3.3 GB, 228 569 images Hours of API requests with Dataset. The result also differs if users edited labels in the meantime.
The fixed dataset rules: HoldoutDenominator 5, Seed 0, ValidateWeight 0.1, Offset 1, Data_2025.05.27.tsv, LegacyCutoff 2025-05-23 — Changing any of them changes the split or the clean subset, and old and new gates stop being comparable.
yolo26x.pt 119 MB Needed only for another fresh candidate (§3).

New data for the next run

  • New labels: buildings whose year built users entered since the last build. Run the Year Built building-data update first.
  • New imagery in orto_datas for already-labelled buildings is not picked up by an append, because the manifest already marks those buildings complete. The same applies to edited labels. If many of either changed, build a new dataset (§3a, new folder) and train both candidates on it.

4b. Sample: continue from the deployed model, with new labels appended

The dataset step appends new buildings to the existing folder (Resume: true; buildings the manifest names are skipped). The training then runs on the whole dataset, old and new. Training on new buildings only would make the detector forget what it learned from the old ones.

D:\YOLO\runs\train10_continue.YOLOTrainingRunOptions.json:

{
  "DatasetOptions": {
    "CountyIds": [5, 204, 310, 314],
    "OutputDirectory": "D:\\YOLO\\train9_dataset",
    "CountOnly": false,
    "Resume": true,
    "LegacyReferencesFilePath": "C:\\ProgramData\\DiGi\\DiGi.GIS.PostgreSQL.UI\\extensions\\DiGi.GIS.YOLO.UI.ConsoleApp\\Data_2025.05.27.tsv",
    "LegacyCutoff": "2025-05-23T00:00:00+00:00",
    "Offset": 1,
    "ValidateWeight": 0.1,
    "HoldoutDenominator": 5,
    "Seed": 0,
    "ModelPath": "C:\\ProgramData\\DiGi\\DiGi.GIS.PostgreSQL.UI\\extensions\\DiGi.GIS.YOLO.UI.ConsoleApp\\YOLO\\models\\model.pt",
    "ReportsDirectory": "D:\\YOLO\\reports",
    "WeightsPaths": [
      "C:\\ProgramData\\DiGi\\DiGi.GIS.PostgreSQL.UI\\extensions\\DiGi.GIS.YOLO.UI.ConsoleApp\\YOLO\\models\\model.pt"
    ]
  },
  "StartWeightsPath": "C:\\ProgramData\\DiGi\\DiGi.GIS.PostgreSQL.UI\\extensions\\DiGi.GIS.YOLO.UI.ConsoleApp\\YOLO\\models\\model.pt",
  "RunName": "train10_continue",
  "ProjectDirectory": "D:\\YOLO\\runs",
  "Epochs": 100,
  "Patience": 50,
  "ImageSize": 640,
  "Batch": 16,
  "Seed": 0,
  "Device": "0",
  "AutoResumeCount": 10,
  "ResumeTraining": false,
  "Steps": ["Dataset", "LabelCheck", "Train", "Validate", "Evaluate"]
}
  • StartWeightsPath is the model to continue. It can be any .pt detector, for example a staged candidate …\YOLO\models\model_train9_fresh.pt. The file is only read, never changed.
  • ImageSize, Batch and Seed stay as the model was trained (the values are in its train_args / args.yaml). Epochs can be lower than for a fresh model, because the model starts from a good point: train9_continue used 100 against fresh's 150.
  • The architecture is the start model's own. Continuing a YOLOv8x model stays YOLOv8x; continuing YOLO26x stays YOLO26x.
  • RunName must be new. The runner refuses a name whose folder exists, and refuses model.
  • Without new labels, drop Dataset (and LabelCheck) from Steps. The run then trains on the existing dataset as it is.

4c. What to expect from a continued run

  • Val mAP50-95 starts close to where the model already was, and climbs slowly: train9_continue went from 0.879 (epoch 1) to 0.902 (20), 0.914 (40), 0.927 (80) and 0.930 (best, epoch 95).
  • On the Test split, train9_continue improved on its own start model (train8) from mAP50-95 0.813 to 0.930, and clean first-detection-year MAE from 2.235 to 1.883 years.

5. Interrupted runs: stop, resume, automatic resume

A multi-day run will be interrupted. train9_continue was interrupted five times in 44 h of training: a power cut, two silent data-loader deadlocks, an NVIDIA driver fault and a native crash in two data-loader workers. Nothing was lost beyond at most one epoch each time, because weights\last.pt is saved after every epoch.

  • Automatic resume (AutoResumeCount, default 0; use about 10 for a long run):

    • The runner ends a training whose output stays silent for InactivityTimeout (15 min by default) and reports it as stalled.
    • It resumes a stall or a crash by itself, up to the count. Before each resume it copies last.pt to weights\last_autoresume<N>_<time>.pt.
    • The log names each resume with its epoch, and a crash is logged with its traceback.
    • A stop requested by the operator is never resumed automatically. train9_fresh was resumed once this way, within 5 s.

  • Operator resume: after a reboot, a power cut, a closed tray or a stop, start the same run again with "ResumeTraining": true and the same RunName and ProjectDirectory:

    {
  "DatasetOptions": {
    "OutputDirectory": "D:\\YOLO\\train10_dataset",
    "WeightsPaths": ["C:\\ProgramData\\DiGi\\DiGi.GIS.PostgreSQL.UI\\extensions\\DiGi.GIS.YOLO.UI.ConsoleApp\\YOLO\\models\\model.pt"]
  },
  "RunName": "train10_fresh",
  "ProjectDirectory": "D:\\YOLO\\runs",
  "Device": "0",
  "AutoResumeCount": 10,
  "ResumeTraining": true,
  "Steps": ["Train", "Validate", "Evaluate"]
}
    • The run continues at the next epoch in the same folder, and the copy, validation and evaluation then run as usual.
    • Steps must include Train and must not include Dataset.
    • Epochs, Patience, ImageSize, Batch, Seed and StartWeightsPath are ignored, because the checkpoint restores them.
    • The tray detects such a run by its name and offers Interrupted after epoch N of M - resume at epoch N+1?. Save the resume's options under a new file name (the tray writes <RunName>.resume-<time>.YOLOTrainingRunOptions.json) so that the original run's file is kept.

  • Python launched by hand, outside the runner, needs two things:

    • pythonw instead of python. Closing the console a Python process is attached to aborts it (forrtl: error (200) … window-CLOSE event).
    • An if __name__ == "__main__": guard in the script. Windows data-loader workers re-import it.

A resumed run is not bit-identical to an uninterrupted one. The results say it was resumed.

6. The gate and shipping a model

The run ends with a table such as train9's:

Weights Test mAP50-95 All Test MAE (n 4 956) Clean MAE (n 179)
train8 (then model.pt) 0.813 0.503 2.235
train9_continue 0.930 0.338 1.883
train9_fresh (selected; model.pt since 2026-10-08) 0.934 0.353 1.855
  • Rule: among the candidates that are not worse than the incumbent on Test mAP and on clean MAE, select the one with the lowest clean MAE. On a tie, select the fresh candidate, whose provenance is fully known. If no candidate qualifies, keep the incumbent.
  • The clean subset is the Test buildings that no model has trained on (Test ∧ ¬Legacy). With 179 buildings it resolves differences of about 0.1–0.15 years.
  • Always compare within one evaluation run: put every incumbent in WeightsPaths. The Test split grows as labels are added, so numbers from different runs are not comparable.
  • Never replace model.pt directly. The runner never does, and neither should an operator. Dating reads the detection columns stored in building_data, so new weights only change how buildings are dated once their counties have been rewritten with the new detections — the every-county rewrite of #23 — and until then a score-only run would date buildings from the old detector's series while the deployed weights claim to be the new one. A detector therefore ships together with the rewrite of its counties' detections: the staged roll-out of #12 Part 4 paired the weights with rewritten detections (and a retrained regressor, now the parked evaluation baseline), see Training the Year Built Model.
    • Until then, keep the candidate in the training directory as model_<RunName>.pt, not beside the deployed model.pt. Only model.pt is deployment content.
    • When it ships, keep the old model as model_<old>.pt in the training directory, record the provenance in the DiGi.YOLO README, re-export the ONNX model, and run the county rewrite before the new detector dates anything.
    • train9 shipped this way on 2026-10-08. The labelled buildings' detections were rewritten with ReferencesFilePath (a few hours instead of weeks), the regressor was retrained on them (DiGi.GIS.ML#13), then both were deployed. The every-county rewrite follows in #23.

7. Traps seen in train9

Trap Effect Avoid it by
Another GPU job during training (LM Studio, a GPU [LongFact]) VRAM taken; one stall coincided with a test training Keeping the GPU to the run. The training facts now refuse to start beside a running training.
Closing the tray, sleep, a Windows Update restart The run stops Disabling sleep and postponing restarts. Resume (§5) afterwards.
AutoResumeCount 0 on a long run A stall waits for an operator: 47 min, 7 h and 1 h 20 min were lost this way Setting about 10.
Reusing a run name for a validation-only or evaluation-only run The earlier run's options file was overwritten (fixed in the tray by DiGi.GIS.PostgreSQL.UI#20) Leaving the run name empty for such runs.
An append after labels were edited Edited labels are not picked up Building a new dataset.
Changing HoldoutDenominator or Seed Test buildings move into Train, and gates are no longer comparable Never changing them.
train_args.model of a resumed run It names last.pt, not the real start weights Recording the start file and its SHA-256 from the run log in the provenance.

Related

Clone this wiki locally