v1.12.0: One prediction API and INT8 exports

RF-DETR 1.12.0 gives native and exported models one prediction API, adds INT8 exports, and speeds up inference on CUDA and Apple Silicon.

• Predict from images, folders, video files, cameras, and RTSP streams through one RFDETRInference API for native and exported models.
• Export INT8 engines for TensorRT, ONNX, and OpenVINO with quantization="int8".
• Get faster default inference: CUDA graphs on CUDA, inductor on Apple silicon, and batches prepared while the last one runs.
• Set a confidence threshold per class with ClassThresholds.

Eight behavior changes need a look, mostly RFDETR.inference() returning a predictor and stricter Kornia training settings; see the Migration guide. Unknown predict() keywords and TRTInference are deprecated.

✨ Spotlights / highlights

One prediction API for native and exported models

RFDETRInference runs a live model, a native checkpoint or any supported export through the same preprocessing and decoding, and returns Supervision detections. It reads directories, globs, video files, cameras, RTSP and .streams manifests, and stream=True yields results as they arrive. (#1563, #1577)

from rfdetr import RFDETRInference, RFDETRSmall

predictor = RFDETRSmall().inference()  # independent, compiled snapshot
detections = predictor.predict("image.jpg")

onnx = RFDETRInference("output/inference_model.onnx")
for result in onnx.predict("video.mp4", stream=True):
    ...

Performance per hardware

RFDETRSmall, batch 1, end to end (preprocessing, forward, postprocessing) through RFDETRInference or the inference() snapshot. Compare within a row, not across machines.

Hardware	Fastest route	Precision	Latency [ms]	FPS [img/s]
NVIDIA L4	TensorRT .trt engine	fp16	3.57	280
Apple M4 Max (40-core GPU)	inference() inductor	fp16	10.3	97.4
x86 CPU (Colab, 4 cores / 8 threads)	OpenVINO IR	fp32	373.7	2.7

On the L4 the inference() CUDA graph in fp16 reaches 3.60 ms, level with the engine at batch 1. On the M4 Max inference() inductor in fp16 runs 10.3 ms against 20.0 ms for eager predict(). On x86 the run-to-run spread is 10 to 17% of the mean, so gaps under about 100 ms are noise. Results for NVIDIA T4, A100 and RTX PRO 6000 and for all four model sizes are in the "Measured Performance by Hardware" section of the export docs index; the mobile and edge CPU tables are pending a rerun.

INT8 exports for TensorRT, ONNX and OpenVINO

quantization="int8" with calibration_data builds INT8 TensorRT engines for detection, segmentation and keypoint models, and static INT8 for ONNX and OpenVINO. On an RTX 5070 and a Tesla T4, TensorRT INT8 ran 1.08× to 1.41× faster than FP16 and was 25% to 39% smaller, at a cost of 0.27 to 1.01 AP points at batch 1. (#1595, #1627, #1560)

model.export(format="tensorrt", quantization="int8", calibration_data="path/to/images", batch_size=8)

Faster default inference

RFDETR.inference() now captures a CUDA graph on CUDA and compiles with inductor on MPS. CUDA prediction prepares the next batch while the current one runs and decodes baseline JPEG batches with nvjpeg. In one benchmark (L4, RF-DETR Nano, FP16, batch 1), CUDA graph replay cut median end-to-end latency from 9.719 ms to 3.884 ms with identical COCO AP. (#1547, #1668, #1670, #1672)

Per-class confidence thresholds

Name the classes that need their own bar and give the rest a default. (#1635)

from rfdetr import ClassThresholds

detections = model.predict(image, threshold=ClassThresholds({"person": 0.35, "traffic light": 0.6}, default=0.5))

Smaller Plus models and a backbone registry

RFDETRAtto, RFDETRFemto and RFDETRPico ship through rfdetr_plus (pip install "rfdetr[plus]", Platform Model License 1.0: commercial use is included when deploying on Roboflow, and commercial self-hosting needs the Enterprise add-on). rfdetr.models.backbone.register_backbone lets an extension package supply a non-DINOv2 encoder. (#1568)

🔄 Migration guide

Each change below failed or behaved differently on 1.11.x with no deprecation shim.

RFDETR.inference() returns a predictor

The return value is now an independent RFDETRInference, and the inplace keyword is rejected. (#1563)

# Before (1.11.x)
model.inference(compile=True, inplace=True)
detections = model.predict(image)

# After (1.12.0)
predictor = model.inference(compile=True)
detections = predictor.predict(image)

Each predictor owns a copy of the current base weights; delete it when you no longer need its runtime. The default compile backend now follows the device: a CUDA graph on CUDA, inductor on MPS, TorchScript elsewhere. Pass compile_backend="torchscript" for the previous default.

Kornia training rejects padded batches

Detection and segmentation training with a Kornia-resolved augmentation_backend raises ValueError during setup when square_resize_div_64=False or multi_scale="per-sample". (#1680)

# After: pick one
model.train(..., square_resize_div_64=True, multi_scale="per-batch")  # or "off"
model.train(..., augmentation_backend="torchvision")  # or "albumentations"

Kornia Affine configs

Affine(translate_px=...) is now applied; 1.11.2 ignored it. Combinations Kornia cannot reproduce (rotation, scale, shear, a nonzero fill, reflected borders) raise ValueError when the pipeline is built, so use augmentation_backend="cpu" (Albumentations, rfdetr[augment]) to keep them. (#1607)

An Affine config without rotate no longer rotates images; it used to sample a random ±15° rotation. Set rotate=(-15, 15) to keep the old behavior. The built-in presets are unaffected. (#1679)

Core ML float16 outputs

format="coreml" with coreml_precision="float16" now returns float32 output arrays, because the .mlpackage is converted as an iOS 15 program. Read outputs as float32. coreml_precision="float32" bundles are unchanged. (#1604)

rfdetr[onnx] drops polygraphy and onnx_graphsurgeon

A host that exports to TensorRT or TFLite with rfdetr[onnx] plus a separately installed tensorrt or onnx2tf must install the missing package itself. (#1589)

pip install polygraphy onnx_graphsurgeon

Segmentation eval outputs

In eval mode, a segmentation LWDETR no longer returns pred_masks in aux_outputs or enc_outputs. Code that runs SetCriterion on eval-mode outputs must pass all_layer_masks=True; the final pred_masks are unchanged. (#1628)

RFDETR.export(batch_size=...) is validated first

batch_size must be an integer. A bool, 2.0, "2" or None now raises ValueError before the forward pass; a bool used to export batch size 1. (#1556)

Deprecations and removals to act on

• Unknown keyword arguments to RFDETR.predict() are ignored with a FutureWarning and will raise TypeError in v2.0.0. stream, batch, vid_stride and stream_buffer raise TypeError now; use RFDETRInference for them. (#1563, #1577)
• TRTInference and OpenVINOInference are deprecated and scheduled for removal in v1.15.0. Use RFDETRInference on the exported artifact. OpenVINOInference now defaults inference_precision to "f32" on every device except NPU; pass inference_precision=None to restore the device default.
• The rfdetr.datasets.aug_config module is removed after deprecation since v1.9.0. Import the presets from rfdetr.datasets.aug_configs. (#1662)

📝 Notable changes

🚀 Added

• Unified inference — public RFDETRInference, versioned export metadata (embedded in ONNX, <artifact>.rfdetr.json sidecars elsewhere), RFDETR.write_export_metadata for older exports, and runtime_options for per-format runtime settings. (#1563)
• Streaming and expanded inputs — RFDETRInference.predict() reads videos, directories, globs, CSV and .streams manifests, cameras, RTSP and HTTP live sources, YouTube and the screen, with stream, batch, vid_stride and stream_buffer; RFDETR.predict() accepts any iterable of images and a BCHW tensor as one batch. New rfdetr[stream] extra. (#1577)
• Per-class thresholds — predict(threshold=ClassThresholds(...)). (#1635)
• TensorRT — INT8 engines for detection, segmentation and keypoint models with SmoothQuant and MLP bias correction on detection; trt_metadata=True writes an <engine>.json description; trt_hardware_compatibility and trt_version_compatible build engines other GPUs or TensorRT releases can load; trt_timing_cache reuses kernel timings; TRTInference(cuda_graph=True) replays the launch from a CUDA graph. (#1595, #1627, #1638, #1642, #1645, #1581, #1580, #1579, #1583)
• Static INT8 for ONNX and OpenVINO with quantization="int8" and calibration_data. (#1560)
• Core ML — notes and the RF-DETR version stored in .mlpackage metadata; coreml_neural_engine=True moves the model onto the Neural Engine (resample and cast still run on the CPU). (#1609, #1586)
• Runtime helpers — public rfdetr.export.inference (TRTInference, load_executorch_method, preprocess_to_nchw, decode_detections, DecodedDetections) and rfdetr.export.benchmark (measure_latency, measure_memory). (#1585)
• RFDETR.inference(compile_backend="cudagraph") — direct CUDA Graph replay for fixed batch size and resolution. (#1547)
• Plus models and backbones — RFDETRAtto, RFDETRFemto, RFDETRPico; register_backbone; ModelConfig.dim_feedforward; RFDETRModelModule(load_encoder_weights=False). (#1568)
• Training — Kornia augmentation for keypoints (#1606); eval_backend="hotcoco_streaming" (#1590, #1614); build_trainer(extra_callbacks=[...]) (#1629).
• RFDETR.predict(antialias=...) — opt-in antialiased resizing. (#1525)

⚠️ Breaking Changes

• Kornia training — padded batches (square_resize_div_64=False, multi_scale="per-sample") raise ValueError. Use square_resize_div_64=True with multi_scale="per-batch" or "off", or augmentation_backend="torchvision" or "albumentations". (#1680)
• RFDETR.inference() — returns an independent RFDETRInference; inplace is removed. Capture the return value and call predictor.predict(...); pass compile_backend="torchscript" for the previous default backend. (#1563)
• Core ML — float16 bundles return float32 outputs. Read float32, or cast with astype(np.float16). (#1604)
• rfdetr[onnx] — no longer installs polygraphy or onnx_graphsurgeon. Run pip install polygraphy onnx_graphsurgeon for TensorRT or TFLite exports. (#1589)
• Kornia Affine — translate_px is applied, unsupported combinations raise, and an unset rotate no longer rotates. Use augmentation_backend="cpu" for unsupported combinations, and set rotate=(-15, 15) for the old rotation. (#1607, #1679)
• Segmentation eval — no pred_masks in auxiliary outputs unless all_layer_masks=True. Pass it when you compute a loss on eval-mode outputs. (#1628)
• export(batch_size=...) — non-integer values raise ValueError. Pass a positive integer such as batch_size=1. (#1556)

🌱 Changed

• Faster inference defaults — CUDA inference() captures a CUDA graph; MPS compiles with inductor, fuses F.grid_sample (and folds LayerScale unless the inductor backend is used); CUDA prediction on an owned model prepares the next batch ahead; baseline JPEG batches decode with nvjpeg; ONNX Runtime on CUDA preprocesses on the GPU with IO binding. (#1672, #1668, #1666, #1670)
• Batching — predict() without batch runs a directory or glob in batches of eight on models without a fixed batch size and on dynamic-batch exports; pass batch=1 for the previous results. (#1664, #1672)
• Host copies — non-CUDA accelerators copy boxes, scores and labels once per batch. (#1664, #1668)
• Training speed — Triton-fused clip, AdamW and EMA for single-GPU compiled BF16 (#1529); vernier>=0.6.0 as the default eval backend (#1661); GPU RLE encoding of segmentation masks (#1615); last-layer-only masks in segmentation eval (#1628); depthwise convolutions use cuDNN on compute capability 8.0 and newer (#1513); the progress bar shows the running mean of loss terms (#1625).
• Dependencies — hotcoco>=1.2.1,<1.3, rfdetr_plus>=1.1.0, ultrafast-pycocotools>=0.1.13. (#1590, #1614, #1568, #1584)
• Export — arguments and packages are checked before the forward pass; LiteRT samples deformable attention from pixel rows; ExecuTorch XNNPACK keeps attention in the delegate; TensorRT refuses a non-int8 quantization or stray calibration_data. (#1556, #1587, #1601, #1595)
• Weights — pretrained weights download from repo.roboflow.com/rfdetr. (#1600)
• Deprecated-model warnings now name the replacement class. (#1618)

🗑️ Deprecated

• Unknown keyword arguments to RFDETR.predict(); removal as TypeError in v2.0.0. (#1563)
• TRTInference and OpenVINOInference; removed in v1.15.0. (#1563)

❌ Removed

• rfdetr.datasets.aug_config, deprecated since v1.9.0. Import from rfdetr.datasets.aug_configs. (#1662)
• onnxsim, onnx_graphsurgeon and polygraphy from the [onnx] extra, and polygraphy and onnxsim from [tflite]. (#1589, #1563)
• The private OnnxOptimizer, onnx_simplify and _onnx.symbolic modules. (#1589)

🔧 Fixed

• Prediction: keypoint covariance on frames with no detections (#1681); bfloat16 tensors as source images (#1682); inference() on CUDA segmentation models under deterministic mode (#1673); keypoint scores in fp16 and bf16 (#1632).
• Training and checkpoints: from_checkpoint("run.pth") loads the file it was given (#1683); a checkpoint-derived num_classes no longer blocks dataset alignment (#1674); resuming applies a changed lr or weight_decay (#1619); BestModelCallback honors the Lightning checkpoint lifecycle (#1678); an invalid better in BestMetricSingle raises ValueError (#1608).
• Augmentation: segmentation boxes follow the transformed masks and empty masks are dropped (#1658); Kornia keeps continuous box edges (#1621); torchvision keypoint flips use continuous coordinates (#1597).
• Export: tflite converts with onnx2tf 2.4.3 on a fresh install (#1563); Core ML segmentation masks line up on the Neural Engine (#1657); FP16 sigmoid in the shared export decoder (#1648); output_name keeps dots (#1641); metadata type checking no longer fails on Windows (#1640).
• Evaluation and data: mixed keypoint-count mAP and mAR (#1686); COCO polygon parts with fewer than three points are skipped (#1644); per-class table class names print verbatim (#1598); evaluate() no longer downloads encoder weights (#1568).
• Loading: no DINOv2 skip warnings on pretrained load (#1653); the class-count warning wording is clearer (#1654); a TorchScript error falls back to eager losses (#1649); the CLI parses the shipped configs and builds the trainer (#1629); the sine-embedding frequencies build in float32 for half-precision Inductor compiles (#1616).

🏆 Contributors

• Lương Thanh Hoàng Phú (@lthphuw, LinkedIn) — built INT8 TensorRT engines for detection, segmentation and keypoint models with quantization="int8"; added hardware- and version-compatible TensorRT engines, CUDA-graph launch replay and kernel-timing reuse; added one threshold per class in predict.
• Lee Clement (@leeclemnet, LinkedIn) — unified native and exported model prediction behind one inference path; expanded prediction inputs and added streaming; added coreml_neural_engine to run Core ML exports fully on the Neural Engine.
• Jesús Royeth (@JESUSROYETH, LinkedIn) — added direct CUDA Graph replay for inference; fused AdamW and EMA updates with Triton; added Kornia augmentation support for keypoints.
• Raghav Rathi (@raghav-rathi, LinkedIn) — encoded segmentation masks to COCO RLE on the GPU; computed only the last decoder layer's masks in segmentation models; fixed the CLI to parse the shipped configs and build the trainer correctly.
• Barsam Ershadi (@BarsamErshadi, LinkedIn) — fixed from_checkpoint to load weights from the file it was given; fixed bfloat16 source images and keypoint covariance on empty frames in prediction; rejected padded Kornia batches for every model.
• Luís Condados (@Gabriellgpc, LinkedIn) — sped up the RF-DETR-Seg training step.
• Matvei Popov (@Matvezy, LinkedIn) — added a backbone registry and Plus model exports for the smaller RF-DETR sizes.
• Erle Carrara (@ecarrara) — switched pretrained weight downloads to the Roboflow repository; cached pretrained weights across CI jobs.
• Tamil Adhavan S K (@adhavan18, LinkedIn) — added opt-in antialiased resizing in predict.
• Yonghye Kwon (@developer0hye, LinkedIn) — sped up ufcoco evaluation by loading box-only detections from one array.
• Noé Fontana (@NoeFontana, LinkedIn) — moved training to vernier 0.6 for faster evaluation.
• Vlad Voropaev (@voropaevv, LinkedIn) — made BestModelCallback honor the Lightning checkpoint lifecycle; added MPS threshold and output-alignment test coverage.
• Christoph Deil (@cdeil, LinkedIn) — preserved continuous box edges in Kornia augmentation.
• Ali Tharwat (@Ali-Tharwat, LinkedIn) — recomputed segmentation boxes from transformed masks in augmentation.
• A Aswanth Raj (@aswanth-07, LinkedIn) — preserved half-precision keypoint trace scores; fixed Windows export metadata type checking.
• Raphael Fakhri (@RaphaelFakhri, LinkedIn) — skipped COCO polygon parts with fewer than three points.
• Mikhail (@mikeedi) — computed FP16 sigmoid in export with a higher-precision buffer.
• Mohammad Hijjawi (@MohammadHijjawi97, LinkedIn) — kept dots that are part of an export output_name; printed class names verbatim in the per-class metrics tab.
• JANG BYUNGKUN (@8rulerstar) — silenced DINOv2 weight warnings when loading pretrained RF-DETR; clarified the message when a set num_classes is kept over the checkpoint's.
• @TurboJonte — stopped a checkpoint-derived num_classes from blocking the dataset's class count.
• Raashish Aggarwal (@raashish1601) — rejected an invalid better value in BestMetricSingle.
• @MaverickTopG — fixed keypoint horizontal flip to use continuous coordinates.
• Erik (@Erol444, LinkedIn) — documented what Apache 2.0 and PML 1.0 mean for commercial use.
• @CodewithGuptaShubh — fixed outdated development setup instructions in the docs.
• Jirka Borovec (@Borda, LinkedIn) — added static INT8 for ONNX and OpenVINO and public runtime paths for exports; made prediction faster with ahead-of-time CUDA batch preparation, GPU JPEG decoding and MPS defaults; expanded hotcoco_streaming as an eval backend; wrote the how-it-works explainers, reorganized export docs and cookbooks, and prepared this release.

Automated contributions: @copilot-swe-agent, @pre-commit-ci

---

Full changelog: 1.11.0...1.12.0
Co-authored with /oss:release.