⊞ Native YOLO-Master GUI Runner with ONNX/ncnn/MNN backends for Windows 10/11

⊞ For Windows 10/11: YOLO-Master Windows Runner (GUI) 1.0.0

An on-device YOLO-Master object detection and instance segmentation for Windows users, and accelerated via NVIDIA CUDA, Vulkan, or OpenCL. It infers on-device with CPU/GPU and nothing leaves the PC.

This is the first public release, and the companion to the  macOS Core ML Runner.

✨ Features

• Three Backends in One App: ONNX Runtime, ncnn, and MNN all ship in the same prebuilt binary. Switch backend from the sidebar and compare on the same image: the preprocessing and decode path is shared, so results match across all three.
• Detection & Segmentation: Runs both bounding-box detectors and instance-segmentation models (the same as the MacOS Core ML Runner).
• Images, Video & Live Camera: Single images, whole-folder batches, and MP4/AVI/MOV/MKV video, plus a low-latency live webcam mode with a real-time FPS / ms-per-frame HUD and a mirror toggle (the same as the MacOS Core ML Runner).
• Real-Time Tuning: Confidence, IoU (NMS), box style, and labels redraw instantly; the forward pass is cached, so tuning never re-runs inference. Letterbox vs. stretch preprocessing is also switchable (the same as the MacOS Core ML Runner).
• Two-Phase Pipeline: Folders and videos are inferred once with a progress bar, then browsed and scrubbed at full speed with the tuned parameters: a 30 fps clip plays back at 30 fps (the same as the MacOS Core ML Runner).
• CPU + GPU: One switch. ONNX runs on CUDA, ncnn on Vulkan, MNN on OpenCL, all in FP16 on the GPU; every backend falls back to CPU cleanly and disaplays the detailed error if a GPU backend is unavailable.
• Image Browser: Folder batches get a thumbnail grid or list view with a resizable icon size, and arrow-key navigation.

🚀 Performance

Live camera inference spoeed (higher = better) on NVIDIA RTX 5070Ti Laptop (Performance Mode), v0.1-seg-N at 640px:

Backend	Device	Infer	FPS
ONNX Runtime	CUDA	9.2ms	~108
MNN	OpenCL	19.7ms	50.8
ncnn	Vulkan	26.0ms	38.5
ONNX Runtime	CPU	41.6ms	24.0

**The runner can perform segmentation at real-time on a consumer laptop GPU. **

🖥️ Demo Screenshot

📥 Installation

Two builds are attached. Both are self-contained: unzip anywhere and run — no installer, no admin rights, no dependencies to download.

Pre-built Bundle	Size	Use Cases
YOLO-Master-Windows-1.0.0.zip	77.8 MB	You want the small download. GPU inference via ncnn-Vulkan / MNN-OpenCL.
YOLO-Master-Windows-CUDA-1.0.0.zip	1.20 GB	You want the fastest path: ONNX on CUDA. Bundles CUDA + cuDNN libraries.

1. Download a zip below and unzip it.
2. Run yolomaster_gui.exe.

The bundled segmentation model loads automatically, so you can open an image straight away.

On first launch, Windows SmartScreen may warn that the publisher is unrecognised: the executable is not code-signed like the Mac one. Click More info → Run anyway. Everything runs locally; the app makes no network requests.

In the CUDA build, the first inference after selecting Device → GPU can take 20 seconds while cuDNN selects convolution algorithms and the driver compiles kernels for your GPU. The window may look frozen; please be patient. Every run after that is super fast.

📦 Pre-exported Models

We ship the following pre-exported models:

model	nc	ONNX	MNN	ncnn
esmoe_n_visdrone	10	✅	✅	✅
esmoe_n_sku110k	1	✅	✅	✅
v0.1_n_visdrone	10	✅	✅	Modifying...
v0.1_n_sku110k	1	✅	✅	Modifying...
v0.1_n_aitodv2	8	✅	✅	Modifying...
v0.1_p2_n_aitodv2	8	✅	✅	Modifying...
uomoe_n_aitodv2	8	✅	✅	Modifying...
uomoe_p2_n_aitodv2	8	✅	✅	Modifying...

Why the six ncnn exports are missing? The v0.1 and UoMoE architectures gate their MoE experts with softmax → torch.topk → torch.gather. However, ncnn has no TopK or gather-by-index layer, so pnnx emits them as unregistered ops and load_param fails outright. Currently we are still attempting to implement TopK + Gather as custom ncnn layers in the runtime (ncnn::Net::register_custom_layer).

💻 Requirements

• Windows 10 (1703 or later) or Windows 11, 64-bit
• 4-core CPU and ≥4GB system memory recommended
• An NVIDIA GPU with a current driver for GPU inference. GeForce RTX 30xx series or newer recommended (Ampere / Ada Lovelace/ Blackwell). Vulkan and OpenCL come with the driver; the CUDA build additionally needs a CUDA 12-capable driver (R525+)
• No CUDA toolkit or cuDNN installation required.
• Runs on machines without a discrete GPU too, on CPU

🤝 Acknowledgements

Built as an extension edge tool of YOLO-Master.
We thank Ultralytics, ONNX Runtime, ncnn, MNN, OpenCV, and Dear ImGui for their great work.

Licensed under AGPL-3.0.

🍎 Native YOLO-Master Core ML Runner for MacOS

For MacOS: YOLO-Master Core ML Runner 1.0.0

On-device YOLO-Master object detection and instance segmentation for macOS, accelerated via Apple Core ML. A native SwiftUI app that features a pick a model and a source (image, folder, video, or the live webcam) and it infers on-device: no command line, no cloud, nothing leaves your Mac.

This is the first public release.

✨ Features

• Detection & Segmentation: Runs both bounding-box detectors and instance-segmentation models. Masks are anti-aliased (no serrated edges), with a Masks / Boxes / Both overlay toggle.
• Images, Video & Live Camera: Single images, whole-folder batches, and MP4 video, plus a low-latency live webcam mode with a real-time FPS / ms-per-frame HUD and a mirror toggle.
• Real-Time Tuning: Confidence, IoU (NMS), box style, and labels redraw instantly; the forward pass is cached, so it does not require to re-run inference at any time. Letterbox vs. stretch preprocessing is also switchable.
• Two-Phase Pipeline: Folders and videos are inferred once with a progress bar, then browsed, scrubbed, and exported with the tuned parameters.
• Export: Write annotated images or MP4 with the current overlay and style.
• Bundled Default Model: Ships with a segmentation model, so it runs the moment you open it; load any other exported Core ML model at any time.
• LoRA Support: Supports LoRA checkpoints of most YOLO-Master models, including but not limited to the EsMoE and the v0.1 families.

🚀 Performance

Live camera inference speed (ms/frame) on Apple Silicon, .mlpackage models via the Core ML CPU + GPU compute unit:

Model	M1 (MacBook Pro 13")	M3 Pro (MacBook Pro 14")	M4 Max (MacBook Pro 16")
v0.1-seg-N	21.2	19.9	20.0
v0.1-seg-N-LoRA	21.3	22.8	21.0
v0.1-N	19.6	19.4	19.8
EsMoE-N	15.5	15.4	15.2
UoMoE-N	14.9	14.1	14.3
YOLOv12-X	45.0	33.7	30.0

Now you can run an X-scale model on a laptop SoC in real-time.

🖥️ Demo Screenshot

Every model runs comfortably in real time even on the base M1; throughput scales with the Mac's GPU and the selected compute unit.

📥 Installation

1. Download YOLO-Master-CoreML-Runner-1.0.0.zip below and unzip it.
2. Double-click YOLO-Master CoreML Runner.app.

That's it. The app is signed and notarized by Apple. Camera access is requested on first use of Live Camera (processed entirely on-device, no internet access needed).

💻 Requirements

• macOS Sonoma or later is recommended
• Supports both Apple Silicon or Intel
• No dependencies to install; the Core ML backend and default model are bundled

🤝 Acknowledgements

Built as an extention tool of YOLO-Master.
We thank Ultralytics and Apple Core ML / coremltools for their great work. Licensed under AGPL-3.0.

🔖 Future Work

I'm currently building a runner for Windows 10/11 as a refinement of the Windows CPU runner (CLI). It will feature a GUI similar to the MacOS runner. However, I'm currently struggling with CUDA compatibility issues. I'll make an update when the Windows version is ready to ship.

YOLO-Master v0.1-seg-N edge models

Segmentation model (COCO-80, 640px) exported to ONNX / NCNN / MNN for the C++ runtime + Windows GUI. Unzip into models/ so v0.1-seg-n.onnx, v0.1-seg-n.mnn (+ .metadata.yaml), and v0.1-seg-n_ncnn/ sit directly in models/.

YOLO-Master-EsMoE-N Pre-exported Models

📦 YOLO-Master-EsMoE-N — VisDrone Models (v0.1)

Pre-exported YOLO-Master-EsMoE-N detection models, trained on VisDrone (10 classes), for use with the edge C++ runtime. All formats are the same graph; accuracy is validated on 548 VisDrone val images against the PyTorch original (mAP50-95 = 0.2036).

Assets

Asset	Format	Size	mAP50-95	Δ vs PyTorch	Notes
esmoe_n_visdrone_sim.onnx	ONNX (opset 12)	10.9 MB	0.2034	−0.02%	onnxsim-simplified, static 1×3×640×640
esmoe_n_visdrone_ncnn.zip	NCNN (pnnx)	9.4 MB	0.2034	−0.02%	unzip → folder with model.ncnn.param / .bin / metadata.yaml
esmoe_n_visdrone.mnn	MNN	10.8 MB	0.2034	−0.02%	converted from the ONNX via mnnconvert
esmoe_n_visdrone_int8_mixed.onnx	ONNX INT8	5.4 MB	0.1952	−0.84%	mixed-precision (head + attention + MoE-router kept FP32)

Class names, input size, and stride are embedded as model metadata — the runtime configures itself; no dataset YAML needed.

Usage

# ONNX (or INT8) — pass the file directly
yolomaster_edge --model esmoe_n_visdrone_sim.onnx --source path/to/image_or_dir --conf 0.25 --out out

# NCNN — unzip first, then point at the folder
unzip esmoe_n_visdrone_ncnn.zip
yolomaster_edge --model esmoe_n_visdrone_ncnn --source path/to/image_or_dir --conf 0.25 --out out

The backend is auto-detected from the model (.onnx → ONNX Runtime, ncnn folder → NCNN). See the repository's TECHNICAL_REPORT.md for the export pipeline, INT8 quantization methodology, and full parity/latency analysis.

YOLO-Master-EsMoE-N — Edge Inference Pre-built Binaries

🚀 YOLO-Master-EsMoE-N — Edge Inference Bundles (v0.1-edge)

Prebuilt, self-contained C++ edge-inference bundles for YOLO-Master-EsMoE-N
(trained on VisDrone, 10 classes). A universal CLI auto-detects backend, class names, and input
size from the model. The bundles now span the full edge matrix — CPU (ONNXRuntime / NCNN on
Linux & Windows) and embedded GPU (native TensorRT on Jetson Orin).

Verified vs the PyTorch original on 548 VisDrone val images: mAP50 −0.10%, mAP50-95 −0.06% (< 0.5%).

Downloads

Bundle	Target	Install needed	Size
yolomaster_edge-linux-x64.tar.gz	any Ubuntu 22.04+ x86_64 (glibc ≥ 2.35)	none — $ORIGIN-rpath'd, libs bundled	35 MB
yolomaster_edge-win-x64.zip	any Windows 10/11 x64	none — MSVC runtime bundled	52 MB
yolomaster_edge-jetson-orin-jp7.tar.gz	any Jetson Orin (Nano/NX/AGX, sm87) on JetPack 7	none — OpenCV bundled; uses stock JetPack TensorRT + CUDA	13 MB

• x86 bundles are fully standalone (Linux runs with no LD_LIBRARY_PATH; Windows bundles its own VC runtime) and ship both ONNX and NCNN models under models/.
• Jetson bundle bundles OpenCV ($ORIGIN/lib rpath) and ships the FP32 ONNX; TensorRT + CUDA are not bundled — they ship with JetPack 7 and are version-matched on each device. You build the FP16 engine once, on-device.

Run

Linux (x86)

tar xzf yolomaster_edge-linux-x64.tar.gz && cd linux-x64
./yolomaster_edge --model models/esmoe_n_visdrone_sim.onnx --source <img|dir> --out out  # onnx
./yolomaster_edge --model models/esmoe_n_visdrone_ncnn --source <img|dir> --out out       # ncnn

Windows

yolomaster_edge.exe --model models\esmoe_n_visdrone_sim.onnx --source your_own_example.jpg --out out

Jetson Orin (JetPack 7)

tar xzf yolomaster_edge-jetson-orin-jp7.tar.gz && cd yolomaster_edge-jetson-orin-jp7
./build_engine.sh          # once per device — builds models/esmoe_n_fp16.engine (~10–15 min)
./yolomaster_edge --model models/esmoe_n_fp16.engine \
    --source <img|dir> --classes visdrone --conf 0.25 --out out

Backend, class names, and imgsz auto-detect from the model. Flags: --conf --iou --imgsz --multi-label --no-save --quiet (--help for all).

Example

Input	Output

Benchmark (VisDrone val, per-frame inference)

Platform	Backend	infer	FPS
💻 Windows 11 CPU	ONNX (ORT)	37.6 ms	25.4
💻 Windows 11 CPU	NCNN	80.1 ms	12.2
🤖 Jetson Orin Nano Super 4GB	TensorRT FP16 (C++)	27.8 ms	35.7
Linux GPU (H200)	ONNX CUDA (C++)	7.8 ms	~128

🏎️💰 Real-time FP16 inference on a 10W, $250 edge device!

Jetson Orin Nano 4 GB detail (FP16 engine, batch-1, 640×640): GPU compute 27.76 ms
(min 27.38 / p99 27.88); end-to-end 28.7 ms (H2D 0.90 + compute 27.76 + D2H 0.05); C++ runner
~30.8 ms/frame → ~33 FPS streaming. Cold-start first inference ~160 ms.

Jetson accuracy (on-device, VisDrone val, 548 images): FP16 engine mAP50 0.3488 / mAP50-95 0.2029
(−0.46% / −0.34% vs FP32 0.3504 / 0.2036) — near-lossless, matching the desktop ONNX reference to within
0.2% mAP points. Reminder: FP16 beats INT8 on this MoE + area-attention architecture — INT8's attention/head
layers fall back to FP32, making QDQ INT8 both slower (45.4 ms) and −2.9 mAP points, while uncalibrated
--int8 collapses completely (mAP50 0.128). For such kinds of mixed detectors, ship FP16.

Notes

• x86_64 CPU tier: Linux floor is glibc 2.35 (Ubuntu 22.04+); older distros may need a build on an older glibc.
• Jetson engine is per-device. TensorRT ties the .engine to the specific GPU + TRT version — do not copy esmoe_n_fp16.engine between machines; re-run build_engine.sh on each device. One bundle serves the whole Orin family (Nano/NX/AGX).
• 4 GB Nano: the engine build is memory-hungry — build_engine.sh uses --builderOptimizationLevel=3 (sidesteps a TensorRT 10.16 FP16 KTM build bug on Orin) and auto-adds an 8 GB swapfile if none exists. Inference itself needs only ~20 MB and runs comfortably. NX/AGX with more RAM simply build faster.
• JetPack 7 required (CUDA 13.x, TensorRT 10.x, Ubuntu 24.04 aarch64). On older JetPack (CUDA 12 / TRT 8–9) the ABI differs — require to rebuild from source (jetson/21_build_trt_runner.sh).
• Build from source: see README.md; full Jetson build/repro details and the reproduced TensorRT-on-Orin gotchas are in jetson/DEPLOYMENT_LOG.md.