RealTimeObjectDetectionCSGO

Object-detection pipeline for CS:GO / CS2 screenshots: collecting training data, labelling it, training a YOLO26 detector, and measuring what such a detector can and cannot do in real time.

Trained model: fvossel/csgo-player-detection · Dataset: fvossel/csgo-player-detection · Demo recording: latest release — four minutes of the overlay running offline against bots, cropped to the 640x640 region the model actually sees (65 MB, H.264).

⚠ Intended use — read this before running anything

This repo exists to show how an AI-based aim assist works in principle, and to measure it. The engineering worth reading is the perception and latency chain: how you get labelled data cheaply, how you keep training and inference geometry consistent, where the milliseconds go, and how you turn a pixel offset into a relative mouse movement. See What this demonstrates below.

It is explicitly not designed to be used for cheating, and it is not built to be usable for it. Run it only offline, against bots, or on your own server with VAC disabled (-insecure, local sv_cheats setups). Nothing here belongs on a VAC-secured server, in matchmaking, on Faceit, or on any other account you care about.

This is not a warning that the implementation is not yet good enough to get away with. The three reasons below are properties of the approach, not gaps to be closed by a more careful version:

1. The overlay is visible to anti-cheat. It is a layered always-on-top window over the game process. Detecting that class of window is routine.
2. The mouse events are marked as injected. overlay.py moves the cursor with SendInput (MOUSEEVENTF_MOVE) and clicks with MOUSEEVENTF_LEFTDOWN/UP. Windows itself flags synthetic input — a low-level mouse hook sees LLMHF_INJECTED on exactly these events. This is not something the --humanize flag hides; the flag changes the timing of the events, not their origin.
3. The behaviour itself is the signature. Valve's behavioural detection looks at aim statistics over many rounds — reaction-time floors, snap geometry, how often the crosshair lands on a head that was never visually tracked. A model that reacts in single-digit milliseconds produces a distribution no human produces. Randomising delays and radii shifts that distribution; it does not make it human. analyze_trajectories.py exists to show you your own signature, not to help you hide it.

Points 2 and 3 are the ones that matter. A more polished implementation could hide the window; it cannot make injected input stop being injected, and it cannot make a model that reacts in milliseconds produce a human distribution. The --humanize flags in this repo exist to make that measurable, and what they show is that the gap narrows without closing.

So the honest summary is not "this would work but please don't". It is: the perception half is genuinely interesting and reusable, the actuation half is a dead end, and the repo is arranged to make both of those visible.

What this demonstrates

The interesting problems here are the ones a detector faces in any real-time setting; the game is the test bench, not the point.

• Matching train and inference geometry. A 1920x1080 frame scaled to 640 turns a head from 13x17 px into 4x6 px. Cropping at native resolution instead keeps the object at the size the model will actually meet. Nearly every decision in the data pipeline follows from that one constraint (§1a).
• Getting labelled data cheaply. Demos already know where the interesting moments are — a shot fired is a moment somebody thought was worth shooting at. Selecting tick ranges from weapon_fire, rendering them from the shooter's viewpoint, then batch pre-annotating with a large model and confirming by hand turns days of labelling into hours (§5).
• Not lying to yourself about the metrics. A random per-frame split puts near-identical neighbouring frames on both sides and inflates the score. The block-wise split and the persistent holdout exist so that numbers from different runs can actually be compared (§1b).
• Making the class balance a decision instead of an accident. Demos from one side yield mostly enemies of the other; filter_by_class.py drops what cannot help rather than labelling more of the same (§5e).
• Latency engineering. Where the milliseconds go, and what actually removes them: overlapping capture with inference (22 → 84 usable frames/s), then TensorRT (12.7 → 4.3 ms inference) — each measured, with the accuracy cost stated (§4).
• The control problem. Turning a pixel offset into relative mouse counts needs the in-game sensitivity, and a single jump to the target is both inaccurate and obviously synthetic, so the movement is spread over steps with damping. This is the part people usually hand-wave; it is also the part that makes the synthetic-input problem unavoidable rather than incidental.
• Measuring the resulting signature. analyze_trajectories.py turns the trigger log into the statistics behavioural detection looks at. Seeing your own reaction-time floor is the most convincing argument against the whole exercise.

The input path is deliberately naive: SendInput, no concealment, no driver-level tricks. That is not an omission to be filled in. It is where the demonstration stops on purpose, and the reasons in the box above explain why filling it in does not lead anywhere good.

Three components that share only config.py and the data directory: training, capture, editor — plus the overlay as a measurement instrument.

Classes (order and IDs from config.CLASS_NAMES, binding for everything):

ID	Name	Boxes	mAP50-95
0	none	0 — empty placeholder, kept for stable IDs	—
1	ct_body	2816	0.797
2	ct_head	2504	0.608
3	t_body	3240	0.784
4	t_head	2923	0.718

5900 hand-confirmed images, 11 483 boxes. The metrics are from the current model, measured on a persistent holdout of 1204 images no training run has seen.

ct_head is the weakest class throughout, and the reason is worth stating: a head at range is 10-20 px across, so it has the least to work with and the least numeric headroom in the box regression. Every step that improved the overall numbers helped it least.

---

Requirements

Python 3.11+ and an existing venv. Everything runs on Windows.

uv pip install -e .[train,capture,editor,export,record,demos,dev]

The extras are split so that no Torch is needed on a pure gaming machine:

Extra	Purpose	Key packages
train	training and validation	ultralytics, torch
capture	in-game capture, dedup	bettercam, pynput, imagehash
editor	web editor with pre-annotation	fastapi, uvicorn, ultralytics
export	ONNX export	onnx, onnxslim, onnxruntime
record	overlay video recording (§4)	opencv-python
demos	tick selection from demo files (§5)	demoparser2

bettercam instead of dxcam: dxcam has been unmaintained since 2022 and crashes under Python 3.13 with current comtypes on the first grab() with an access violation. bettercam is a maintained fork with the same API; capture.py automatically falls back to dxcam if only that one is present.

---

1. Training pipeline

Getting the data first

No image data ships with this repository. data/ is gitignored apart from data/splits/holdout.json, so a fresh clone has no data/train, no data/val and no data/crops640. That is deliberate: the frames show Counter-Strike and are derivative of Valve's assets, and the original starting dataset states no licence that would let it be redistributed here.

Three ways to get to a trainable dataset, in ascending order of effort:

Route	What you get	Effort
The published dataset	5473 labelled crops, ready to look at	download, then convert
Record your own (§2, §3)	whatever you play	hours of clicking
Harvest from demos (§5)	thousands of frames from existing matches	the fast route

The published dataset is fvossel/csgo-player-detection. It is in Hugging Face imagefolder layout — images plus a metadata.jsonl with COCO boxes — not the images/ + labels/*.txt YOLO layout that train.py reads, so it needs converting first. There is no converter in this repo yet.

The original full-frame dataset §1a expects under data/train and data/val is Counter Strike 2 Body and Head Classification on Kaggle. It states no licence, so check for yourself what you may do with it.

If you only want to see the model work, none of this is needed: §4 runs against the published weights and your own screen.

1a. Produce crops (once, mandatory before training)

The raw dataset consists of 1920x1080 full frames, but the capture tool delivers 640x640 crops in native resolution. At imgsz=640 on full frames a t_head would shrink from 13x17 px to 4x6 px — training and inference would see completely different object scales. Therefore the crops are cut first:

python scripts/prepare_crops.py --dry-run

python scripts/prepare_crops.py --force

Result: 427 frames into data/crops640/{images,labels}, 969 → 840 boxes (86.7 % keep rate), of which 43 frames without a box are deliberately kept as background images with an empty label file.

The originals under data/train and data/val are left untouched — the script only reads them. Both have to be put there by you first, see Getting the data above; without them this step exits with image folder missing.

1b. Train

python scripts/train.py --epochs 150 --batch 16

Train multiple sources together — base dataset plus newly labeled sessions:

python scripts/train.py --data-dir data/crops640 data/captures/<session> --epochs 150

The directories stay separate. That is intentional: prepare_crops.py --force rebuilds data/crops640 from the originals, so captures copied into it would be gone on the next run. The split's block formation never spans two sources.

Produce only the split and data.yaml without training:

python scripts/train.py --dry-run

What the script does:

• generates data/data.yaml from the folder structure instead of hardcoding it
• splits block-wise instead of randomly per frame (--block-size, default 10): adjacent screenshots come from the same round — screenshot_10 and _11 show the same scene at round time 5:35 vs. 5:33. A random per-frame split distributes such neighbors across train and val and inflates the metrics. --split-mode random exists only to measure exactly this effect.
• writes the split lists to data/splits/{train,val}.txt, so the original folders stay unchanged
• prints instances per class per split before training and explicitly names classes without val instances
• writes per-class metrics to reports/metrics_<timestamp>.json
• copies the best weight to models/yolo26n_csgo_<timestamp>.pt and additionally to models/latest.pt

Classes without validation instances appear in the report as "no validation instances - metric not meaningful", not as 0.0.

Two flags decide which images of a source are used:

python scripts/train.py --confirmed-only --data-dir data/crops640 data/captures/<session>

• --confirmed-only takes only images confirmed in the editor, read from editor_state.json. Use this for any source that preannotate.py has touched, otherwise unchecked model output is trained on and the model learns its own mistakes. Sources without an editor_state.json — data/crops640 — were labeled by hand throughout and are used in full.
• --allow-unlabeled skips images without a label file instead of aborting. For sessions still being labeled, so an interim model can pre-annotate the rest. Unlabeled images are skipped, not treated as empty: a missing file is not the same statement as an empty one.

Persistent holdout

The block-wise split prevents leakage within one run. Across runs it does nothing: every run redraws the split, so an image that trained model A can end up validating model B. Comparing the two then measures how much of B's val set A has memorised. This is not hypothetical — it produced an apparent 0.077 mAP50-95 advantage for an older model that was in fact being scored on its own training data.

data/splits/holdout.json fixes that. It records both sides — which images are held out and which are pinned to training — and an image's assignment is decided the first time it is seen and never revisited:

• a holdout image is never trained on,
• a train image is never validated on.

Both directions are needed. Recording only the holdout leaves the reverse open: on a later run an image that older models trained on could be drawn into the holdout and would then be scoring those models on their own training data.

The holdout grows with the data. It is not a fixed set — what is fixed is each individual image's assignment. Every new session contributes roughly --val-fraction of its own images, so the holdout stays representative as the dataset grows:

holdout: 1204 images (was 680, +524 from new data), fingerprint a1d8b4006d7e

Entries whose file disappeared (discarded in the editor) are pruned.

Two runs may be compared directly only if their fingerprints match — the fingerprint is a hash of the holdout contents and is printed by every run and stored in reports/metrics_*.json under split.holdout. When the holdout grows, the fingerprint changes; the older images keep their assignment, so the two runs remain comparable on the intersection, just not on the raw totals.

The guarantee starts when the registry is created — weights trained before that may have seen some of these images, so comparisons against older .pt files stay unreliable. --reset-holdout redraws from scratch and invalidates comparability with every earlier run; --split-mode random bypasses the registry entirely, since it exists only to measure the leakage effect.

Without a cache every image is read from disk and decoded again in every epoch. --cache ram keeps the decoded images in memory instead:

python scripts/train.py --data-dir data/crops640 --epochs 150 --cache ram

About 1.2 GB for 1000 images at 640x640. This changes wall-clock time only, not the result — the reported metrics stay comparable. --cache disk writes decoded .npy files next to the originals and is only worth it on a slow HDD. Default is off; the setting lands in the report so runtimes stay comparable later.

Expect weak values on the small dataset. A 3-epoch trial run yielded mAP50 0.319 / mAP50-95 0.210 — the script reports that as weak, not as a success.

1c. Export

python scripts/export.py --weights models/latest.pt

python scripts/export.py --weights models/latest.pt --format both

--format both additionally builds a TensorRT FP16 engine. That takes minutes and is tied to the specific GPU. If it fails, the script ends with exit code 2 and still names the produced ONNX — as opposed to exit code 1 (total failure).

---

2. Capture tool

Runs in the background while playing and saves a 640x640 crop around the screen center on every left click, in native resolution without scaling.

python scripts/capture.py

Additionally trigger every N seconds independently of clicks:

python scripts/capture.py --interval --interval-seconds 5

Choosing --interval-seconds smaller than the minimum gap achieves nothing — the script points this out at startup and discards the surplus triggers.

CS2 must run in borderless windowed mode. The Desktop Duplication API returns black or no frames in exclusive fullscreen mode. Many entries under "no new frame" in the final report point exactly to this.

• Only when CS2 is in the foreground. Without this filter every left click triggers system-wide and the dataset fills up with browser and desktop captures. The EXE name of the active window is checked, not the window title — a browser tab named "Counter-Strike 2" would otherwise pass. To find the right name, start this and then alt-tab to the game — it prints every change of the foreground process until you stop it with Ctrl+C:

  python scripts/capture.py --show-active

  A different name via --process csgo.exe, disable via --any-window.

• Minimum gap of 5 s between two captures (--min-gap-seconds). Within a few seconds the player usually stands in the same spot and looks in the same direction — such frames cost labeling time without adding information. If you prefer to collect more and sort out afterwards, lower the value and let scripts/dedup.py clean up.

• F8 pauses and resumes, F9 exits cleanly

• Writing happens in a worker thread with a queue so the capture does not slow down the game. The latency is measured and printed aggregated (measured on an RTX 5070: screen access median ~4 ms, writing ~8 ms in the worker thread)

• Output: data/captures/<session>/images/*.png plus meta/*.json with timestamp, session ID, screen resolution, crop offset and trigger reason

The process sets SetProcessDpiAwareness at startup — without it, with display scaling enabled, Windows reports the logical rather than the physical resolution (at 150 % about 1707x1067 instead of 2560x1600), and the "native" crop would in truth be an upscaled one.

Remove duplicates

python scripts/dedup.py data/captures/<session>/images --dry-run

python scripts/dedup.py data/captures/<session>/images --apply

Perceptual hash (phash), threshold via --threshold (default 5, 64-bit hash). Without --apply nothing is changed. Dropped frames take their sidecar and label files with them so that no orphaned files remain.

---

3. Web editor

Corrects model predictions instead of starting from scratch by hand.

python scripts/editor.py --dir data/captures/<session>

python scripts/editor.py --dir data/crops640 --weights models/latest.pt --conf 0.25

--dir takes several directories, and a directory without an images/ subfolder is treated as a parent whose subdirectories are searched. Both of these open every capture session at once:

python scripts/editor.py --dir data/captures

python scripts/editor.py --dir data/captures/<a> data/captures/<b> data/crops640

Add --recursive to search at any depth instead of only one level down. Each source keeps its own editor_state.json, so a session stays self-contained. Image stems are only unique within a source — two capture sessions both start at frame_0000 — so images are addressed by <source index>/<stem> internally and the sidebar shows the source name next to each file.

Then open http://127.0.0.1:8000. The model is loaded once at startup, not per request. Without a weight or with --no-model the editor starts and opens all images empty.

Expected directory layout:

<root>/images/<name>.png      raw images
<root>/labels/<name>.txt      YOLO labels, exactly 5 columns
<root>/preds/<name>.json      confidence per box, parallel
<root>/editor_state.json      which images are confirmed

The confidence lives deliberately in its own file — a sixth column in the .txt would violate the YOLO format.

Controls

Everything is reachable by both mouse and keyboard.

Key	Effect
Space	confirm and continue — the most common case, one keystroke
← →	navigate
1–4	set the class of the selected box
Tab	select the next box
Del	delete the selected box
Shift+Del	discard the whole image
Ctrl+Z	undo
0	reset view

Discarding an image moves it — together with its label, preds and meta sidecars — into <source>/deleted/, keeping the original folder layout. Nothing is unlinked, so a misclick is undone by moving the files back by hand. That is why there is no confirmation dialog: at one keystroke per image the dialog would cost more than the mistake it prevents.

With the mouse: dragging on an empty area draws a new box, dragging on a box moves it, the handles change its size. The mouse wheel zooms on the cursor, right- or middle-click drag pans the view. In the sidebar you can jump to any image directly and filter by open/done; the buttons in the toolbar cover all keyboard shortcuts.

Saving happens automatically on image change, there is no save button. The editor starts at the first not-yet-confirmed image.

---

4. Real-time overlay & benchmark (research instrument)

Runs the model live on the screen centre and draws, on a transparent click-through window over the game: the detected boxes, the head nearest to the crosshair, the trigger radius, and the path a step-wise aim assist would take. It measures end-to-end latency and logs each trigger for offline analysis.

This section is where the warning at the top applies most directly. With --move-mouse the overlay does not simulate anything — it calls SendInput and moves the real cursor. Offline against bots only.

Needs the capture and train extras (bettercam, pynput, ultralytics); the window uses tkinter from the standard library.

Visualisation only, no input

The default. Draws and measures, emits no input event:

python scripts/overlay.py --weights models/latest.engine

F8 pauses and resumes, F9 quits. The overlay is placed on the primary monitor — for a game on a second monitor the boxes are offset; use --monitor.

Useful flags:

Flag	Effect
--radius 0.2	trigger radius as a fraction of the crop
--target-classes 2 4	which class IDs count as targets, here the two heads
--conf 0.1	detection threshold, lower shows more and worse boxes
--size 640	crop edge length, must match what the model was trained on
--no-log	do not write the trigger log

Recording

python scripts/overlay.py --weights models/latest.engine --record demo.mp4

Writes an annotated MP4 of the crop: the boxes with class and confidence, the trigger radius, the line to the current target, and a HUD with frame rate and latency. Needs the record extra (uv pip install -e .[record]); without it the overlay still runs and only the recording is skipped, with a warning.

This records what the model sees, which is not the same as recording the screen. If you want the game with the overlay on top, use a screen recorder — but in display capture mode, not game capture: the overlay is a separate layered window, not part of the game process, so a game capture produces a video with no boxes in it at all. The two are complementary, and a write-up usually wants both.

Frames are written on a wall-clock schedule (--record-fps, default 30) rather than one per loop pass. The loop runs at whatever rate inference allows and that rate varies, so one frame per pass would give a video whose playback speed silently depends on how busy the GPU was.

With cursor control

Everything below emits real input events. Only offline, against bots, on an -insecure client:

Flag	Effect
--move-mouse	enables SendInput; without it nothing below does anything
--sensitivity	your in-game sensitivity, needed to convert pixels to counts
--boost	multiplier on the computed movement
--damping, --smooth-speed, --steps	how the movement is spread over time
--hard-aim	aim regardless of the trigger radius
--auto-click, --click-threshold, --num-clicks, --click-delay	fire when the crosshair is close enough
--humanize, --reaction-min/max, --jitter, --overshoot, --variable-radius	randomised timing and geometry
--predictive, --adaptive-boost, --instant	aim behaviour variants

On --humanize and friends: they change when and how far the events are sent. They do not change that the events are synthetic, and Windows marks them as such. Treat them as parameters of the experiment, not as concealment.

Headless latency benchmark, no window (answers "is it real-time?"):

python scripts/overlay.py --benchmark 10

The benchmark warms up first (the one-off CUDA cold start is >1 s and would otherwise dominate) and prints which capture mode was used.

Capture runs in a background thread (capture.FrameSource): the grab overlaps with the inference instead of waiting for it. camera.grab() returns nothing while no new frame has arrived, so the old serial loop burned its retry budget and then discarded the whole iteration. Measured against the real screen, same loop, 6.5 ms of simulated inference per pass:

Mode	usable frames	throughput
ondemand (grab() + retries)	43 of 120	22.2 /s
background (ring buffer)	120 of 120	84.4 /s

In the pipeline the read itself drops to ~0.1 ms median (a buffer copy), so inference dominates: on an RTX 5070 roughly 9 ms median inference, ~9 ms end-to-end, i.e. a reaction floor around 110 Hz. Reads always use a timeout — bettercam's own get_latest_frame() waits forever, which would freeze the HUD as soon as the game stops rendering (alt-tab, menu). If the camera does not expose the ring buffer, FrameSource falls back to grab() and says so.

--capture-fps sets the poll rate of that thread, default 1000 Hz. The duplicator uses a 0 ms timeout, so --capture-fps 0 polls without any timer and costs a full CPU core to save under a millisecond. The poll rate trades against inference, because the thread competes for CPU — measured on the same machine:

--capture-fps	capture (median)	inference (median)	sum
60	7.2 ms	7.7 ms	14.9 ms
1000 (default)	0.1 ms	8.9 ms	9.0 ms

TensorRT engine

With capture down to ~0.1 ms, inference is what is left. Build the engine and point the overlay at it — --weights is mandatory, resolve_weights() falls back to models/latest.pt and would never find the engine on its own:

python scripts/export.py --weights models/latest.pt --format engine

python scripts/overlay.py --weights models/latest.engine

Latency on an RTX 5070 Laptop, three overlay benchmark runs each, back to back. Accuracy from model.val() on the 1204-image persistent holdout, batch 1:

	inference (median)	end-to-end	mAP50	mAP50-95	Precision	Recall
latest.pt (PyTorch FP32)	12.7 ms	12.9 ms	0.9053	0.7267	0.9363	0.8407
latest.engine (TensorRT FP16)	4.3 ms	4.5 ms	0.9009	0.7262	0.9388	0.8394

~3x faster for 0.07 % relative mAP50-95. Per class the FP16 difference is ct_body −0.002, ct_head +0.001, t_body +0.003, t_head −0.005 — inside the noise, with no systematic direction. Precision is marginally better in FP16, recall marginally worse.

That was not always the case. On an earlier, much smaller dataset the same comparison cost 2.6 % relative mAP50-95 and hit ct_head hardest, which is the textbook FP16 failure on small objects: the box regression head has the least numeric headroom exactly where the boxes are smallest. With ~4x the training data the penalty disappeared into the noise. Worth knowing if you are deciding whether to quantise a model trained on very little data — measure it, do not assume either outcome.

Re-run the validation after any driver update: the engine is tied to this GPU, this driver and this TensorRT version, and is not portable.

Characterize the aim behavior

The overlay writes one JSONL line per would-be trigger to data/overlay_logs/<session>.jsonl. This script summarizes them into the kind of signature behavioral anti-cheat cares about — reaction-latency floor, engagement rate, and geometry:

python scripts/analyze_trajectories.py data/overlay_logs/<session>.jsonl

No game or model is needed for the analysis — it reads the log only.

The point of this script is to make the signature visible, not to help hide it. A reaction-time floor no human reaches, an engagement rate no human sustains and snap geometry that never overshoots are exactly what behavioural detection looks for, and they show up here in a few hundred lines of log. If your reaction is to tune the randomisation until the numbers look human, re-read the warning at the top of this file: shifting a distribution is not the same as coming from it, and the injected-input flag on every event is unaffected either way.

---

5. Training data from demos

Playing and clicking is the slow way to collect frames. A demo already knows where the interesting moments are, and CS Demo Manager can render arbitrary tick ranges to a lossless image sequence — no video compression on exactly the small heads the model struggles with most.

Needs CS Demo Manager (the installer ships the csdm CLI) and the demos extra (uv pip install -e .[demos]).

5a. Pick the tick ranges

python scripts/demo_sequences.py <replays folder> --shooter-team t --stats-only

python scripts/demo_sequences.py <replays folder> --shooter-team t --max-sequences 120

Selection is on weapon fire, not on kills. Kills are the subset of shots that connected, so selecting on them selects for "enemy well lit, centred, close" — the easy cases. Shots include the misses, the long-range sprays and the half-covered targets, which is the distribution the model actually meets. csdm --mode player --event only offers kills,deaths,rounds, so the ranges are computed here and handed over via --config-file.

--shooter-team t yields frames showing CTs — the ct_head class that is hardest and least represented. A shot only counts if a living enemy is inside the shooter's view cone and within range; consecutive shots merge into one sequence so a 20-bullet spray does not become 20 overlapping clips.

The written config forces showXRay: false. X-Ray draws players through walls — leaving it on would put targets into the image that are not really visible and teach the model to hallucinate enemies behind cover.

5b. Render

csdm video --config-file data/demo_configs/<demo>.json

5c. Import

python scripts/import_frames.py data/demo_configs/rendered/<demo>

Center-crops to 640x640 with the same arithmetic as capture.py — verified identical — and writes a normal capture session, so editor, dedup and training work unchanged.

Two settings decide whether the data is usable at all:

• Render at the deploy resolution (width/height in config.DemoDefaults, default 2560x1600). At 1080p the centre crop covers a different field of view and heads end up at a different pixel size than at inference. import_frames.py warns when it sees a mismatch.
• Frame rate 5, not 30. Neighbouring frames at 30 fps are near-duplicates — the same temporal redundancy the block-wise split exists for. Eight demos yield ~76 000 frames at 30 fps and ~8 400 at 5 fps with --max-sequences 120.

The cone test does not check line of sight, so some sequences show nothing but a wall. That is deliberate: a strict test needs raytracing against the map geometry and would lose exactly the half-covered targets worth training on. The model filters the empties out afterwards.

5d. Pre-annotate a whole session at once

The editor runs the model when an image is opened for the first time, which puts a wait in front of every image. For a few thousand frames that is the wrong place for it:

python scripts/preannotate.py --dir data/captures/<session> --weights models/annot_yolo26x.pt

Writes exactly what the editor would have written — labels/*.txt plus the confidence sidecar in preds/*.json — so afterwards the editor only reads files:

python scripts/editor.py --dir data/captures/<session> --no-model

Confirmed images are never touched, not even with --overwrite. --overwrite re-predicts images that already carry an unconfirmed prediction, which is what you want after training a better annotation model.

Use a bigger model here than for the overlay. Pre-annotation runs once per image and is not on the latency path, so yolo26x is a sensible choice while the overlay keeps the nano model. Train it with --no-copy, otherwise it lands in models/latest.pt and the overlay picks it up.

5e. Balance the classes

Demos recorded from one side yield mostly enemies of the other. Labeling more of the same does not fix that — dropping what cannot help does:

python scripts/filter_by_class.py --dir data/captures/<session> --keep t_body t_head
python scripts/filter_by_class.py --dir data/captures/<session> --keep t_body t_head --apply

The decision is made on the label file, which for unconfirmed images is the model prediction. That is good enough to decide whether an image is worth looking at, even where it is not good enough to train on. Confirmed images are never dropped. Discarded images move to <session>/deleted/ with all their sidecars, so a wrong call is undone by moving them back.

This also removes the frames with no enemy in view — the cone test in 5a does not check line of sight, so roughly 40 % of rendered frames show nothing but a wall.

---

6. Publish the dataset

python scripts/export_hf.py --dry-run
python scripts/export_hf.py

Builds a Hugging Face imagefolder dataset in data/hf_export: one directory per split, each with a metadata.jsonl carrying the boxes as COCO [x, y, w, h] in absolute pixels.

Three decisions worth knowing:

• Only confirmed images. Since preannotate.py exists, a label file no longer proves a human looked at the image.
• The split comes from the holdout registry, not drawn anew, so the published validation split is the one the reported metrics were measured on.
• Images are hardlinked, not copied. Costs no extra space, but the export and the sessions then share the same data — deleting an image in one does not remove it from the other. Use --copy if that bothers you.

Pass --data-dir explicitly to leave a source out, for instance one you have no right to redistribute. The generated dataset card adapts to what is actually in the export.

---

Conventions

• All constants in config.py, no scattered values in the scripts
• All paths relative to the project root
• Type hints throughout, ruff for linting and formatting:

python -m ruff check scripts/ config.py && python -m ruff format scripts/ config.py