YOLO detector training

YOLO detector training

The tray-app task Train YOLO detector (UIYOLOTrainingTask) retrains the building detector that the Predict year built task scores imagery with. It covers the procedure of DiGi-Organization/DiGi.GIS.YOLO.UI#12 in one run: dataset build, label check, training, validation on the Test split, and detector evaluation. Before this task, that procedure was a sequence of console commands and python train.py calls with paths typed by hand.

The run happens in the headless runner DiGi.GIS.YOLO.UI.ConsoleApp --train <options file> (DiGi-Organization/DiGi.GIS.YOLO.UI#15). This is the same executable Predict year built uses, so the task appears in the client tab only where that runner is deployed.

---

1. Before the first run

What	Where	Why
Runner with --train	bin\extensions\DiGi.GIS.YOLO.UI.ConsoleApp\ (checked first), or the runner's workspace bin	A runner built before --train exits with Configuration (Unknown mode '--train').
GIS_WebAPI_Client.conf	beside the runner	The dataset step authorizes with the runner's key, not the tray app's.
model.pt	the runner's user files/YOLO/models/ (deployed as YOLO\models\model.pt)	The Re-train start weights and the incumbent in the gate.
yolo26x.pt	the runner's user files/YOLO/models/base/ (deployed as YOLO\models\base\yolo26x.pt)	The fresh-start checkpoint. Its SHA-256 is in the DiGi.YOLO README.
Data_2025.05.27.tsv	beside the runner (user files/)	The legacy references the dataset build reads.
CPython with ultralytics and torch	on PATH, or named in the dialog	Checked by the tray before launch.

2. The two scenarios

	Re-train	Start from yolo26x.pt
Start weights	model.pt	yolo26x.pt
Epoch ceiling	150	300
Dataset folder	the existing one, with the chosen counties appended (Resume on; the builder is keyed by reference)	a new folder; one that already holds a dataset is refused

Switching scenario in the dialog resets only the start weights and the epochs. Every value can still be edited.

The remaining defaults are patience 50, image size 640, batch 16 and seed 0.

3. A run

1. Start Train YOLO detector from the client tab.
2. Fill in the dialog:
   • Choose the counties. A multi-part county is one entry per piece, so select every piece.
   • Give an absolute dataset directory.
   • Give a new run name, and an absolute project directory that is not inside a YOLO\models folder.
3. Read the confirmation. It names the scenario, the steps, the counties, the start weights, the dataset folder with whether it is appended to or built fresh, and the run folder.
4. The tray then checks everything that can fail before launch, and logs a refusal naming the option. It checks that:
   • the start weights exist as a .pt file
   • the project folder is outside YOLO\models
   • the run name is free and is not model
   • the legacy references file is present when the dataset step is on
   • the label check detector (model.pt) is present when the label check is on
   • every gate weights file is present when the evaluation is on
   • the interpreter can load the start weights. With the interpreter box empty, the one found on PATH that passes this check is written into the options, because the runner refuses training without one
5. The options are written to <ProjectDirectory>\<RunName>.YOLOTrainingRunOptions.json, next to the run folder rather than inside it, because the runner refuses an existing run folder. That file is the record of the run. A resume writes <RunName>.resume-<yyyyMMdd_HHmmss>.YOLOTrainingRunOptions.json instead, so the original run's file is never overwritten.
6. The runner's output is logged line by line. The dataset step reports progress. The exit code is logged by name: Training is 6, Validation is 7.

Stopping the task kills the whole process tree, interpreter included. What earlier steps wrote stays. The dataset manifest lets a Re-train run continue an interrupted build; Start from yolo26x.pt does not resume, so to continue a fresh build that was stopped, choose Re-train and set the start weights back to yolo26x.pt and the epochs to 300. A training that was stopped, stalled or crashed is resumed from the dialog — see §4.

4. Resuming an interrupted run

A training stopped mid-epoch — from the tray, by a crash, or by a stall — is not lost. last.pt holds the epoch, the optimizer state and every argument the run was started with, so the run can continue where it stopped.

When the run name and project directory name a folder that holds an unfinished weights\last.pt and no completed <RunName>.pt, the dialog shows Interrupted after epoch N of M - resume at epoch N+1? (N is the last completed epoch; the runner logs the same resume as "Resuming … from epoch N+1 of M") and offers Resume interrupted run.

• Ticking it locks what a resume cannot change: the start weights, epochs, patience, image size, batch and seed come from the checkpoint, and the dataset build is switched off. Device, interpreter, working directory, gate weights and the steps after training stay editable; the Training step is still required.
• The run continues last.pt at the next epoch in the same folder, then copies, re-checks, validates and evaluates exactly as a fresh run.
• A resumed run is not bit-identical to an uninterrupted one (the dataloader RNG restarts).
• A resume writes its own <RunName>.resume-<yyyyMMdd_HHmmss>.YOLOTrainingRunOptions.json; the original run's file is never overwritten.
• The tray refuses before launch what the runner refuses: the training step not selected, the dataset step selected, a missing folder or last.pt, a completed <RunName>.pt, and a checkpoint that is unreadable, finished, records a missing dataset, or records a project or name that was moved or renamed.

Closing the application or a power cut still stops the run. Resume shortens the recovery; it does not make a run survive an interruption.

Stalls and automatic resume

A training can stop producing output while staying alive — a data-loader deadlock holds the GPU with every process at 0 % CPU — or its process can die with a driver fault. The dialog's Automatic resumes (default 3) and Stall limit (minutes) (empty for the runner's 15) set AutoResumeCount and InactivityTimeout, and are enabled only with the Training step.

• A stall (no output for the limit) or a crash is continued from weights\last.pt, up to the count, each time after copying the checkpoint to weights\last_autoresume<N>_<yyyyMMdd_HHmmss>.pt.
• A stop from the tray, a preflight refusal and a finished checkpoint are never resumed. An operator resume above is unchanged by these settings.
• The runner logs Training stalled at epoch E (last output hh:mm:ss) - automatic resume N of M, or Training exited with code X at epoch E - automatic resume N of M, preceded by the last 30 lines of the crashed attempt's error output ( | …, its traceback), or by The attempt that exited with code X wrote no error output - … when a process ended from outside wrote none. The attempt that ends the run is logged the same way, ending - no automatic resume left (M of M used) (or - automatic resume is off (AutoResumeCount 0)), and the closing Resumed automatically N time(s): Stalled at epoch E (hh:mm:ss), … names the same interrupted epochs, and prints Resumed automatically N time(s): … at the end. A run that exhausts the count fails with the last reason named.

5. Result and gate

• The trained weights are copied to <ProjectDirectory>\<RunName>\<RunName>.pt. The run log records the SHA-256 of both the start weights and the output weights. model.pt is never overwritten.
• The evaluation compares the gate weights (the incumbents named in the dialog) with the new weights, row by row.
• Shipping or holding the new weights is a manual decision made from that table (DiGi-Organization/DiGi.GIS.YOLO.UI#12, Part 3 step 7). To ship them, replace the runner's model.pt by hand, then re-run Predict year built for the counties that need it.

---

Related: DiGi-Organization/DiGi.GIS.PostgreSQL.UI#19, DiGi-Organization/DiGi.GIS.PostgreSQL.UI#21, DiGi-Organization/DiGi.GIS.YOLO.UI#15, DiGi-Organization/DiGi.GIS.YOLO.UI#16, DiGi-Organization/DiGi.GIS.YOLO.UI#17, DiGi-Organization/DiGi.YOLO#16.