Skip to content

feat(backend): rfdetr-cpp native object detection + segmentation backend#10028

Merged
mudler merged 1 commit into
masterfrom
feat/rfdetr-cpp-backend
May 27, 2026
Merged

feat(backend): rfdetr-cpp native object detection + segmentation backend#10028
mudler merged 1 commit into
masterfrom
feat/rfdetr-cpp-backend

Conversation

@localai-bot
Copy link
Copy Markdown
Collaborator

Summary

Adds a Go native gRPC backend (backend/go/rfdetr-cpp/) that dlopens librfdetr.so (built from mudler/rf-detr.cpp at the pinned RFDETR_VERSION) via purego and exposes the rfdetr.cpp inference pipeline through LocalAI's existing Detect RPC.

Supports all 5 RF-DETR detection variants (Nano/Small/Base/Medium/Large) and 3 segmentation variants (SegNano/SegSmall/SegMedium) with F32/F16/Q8_0/Q4_K quantizations. Pre-built GGUFs ship at mudler/rfdetr-cpp-* on HuggingFace (32 models total).

Detection returns Bbox + class_name + confidence; segmentation also returns PNG-encoded per-detection masks via the rfdetr_capi_get_detection_{class_id,box,score,class_name,mask_png} accessor functions.

Wiring (full adding-backends.md checklist)

  • backend/go/rfdetr-cpp/{main.go, gorfdetrcpp.go, CMakeLists.txt, Makefile, run.sh, package.sh, test.sh, .gitignore}
  • Top-level Makefile: BACKEND_RFDETR_CPP, docker-build-rfdetr-cpp, .NOTPARALLEL, prepare-test-extra, test-extra
  • .github/backend-matrix.yml: CPU + CUDA 12 + CUDA 13 + HIP + Vulkan + SYCL f16/f32 + L4T variants
  • backend/index.yaml: &rfdetrcpp meta + 17 image entries (latest + development for each variant)
  • .github/workflows/bump_deps.yaml: RFDETR_VERSION pin tracking against mudler/rf-detr.cpp main branch
  • gallery/index.yaml: rfdetr-cpp-nano + rfdetr-cpp-base model entries
  • core/gallery/importers/rfdetr.go: GGUF auto-routing for HF imports (routes `.gguf`-bearing repos to `rfdetr-cpp`, falls through to Python `rfdetr` backend otherwise)
  • core/gallery/importers/rfdetr_test.go: 6-entry table-driven test + live HF cross-check against `mudler/rfdetr-cpp-nano`

The `scripts/changed-backends.js` registration was checked and is not needed — the existing `endsWith("golang")` branch already routes `./backend/Dockerfile.golang` entries to `backend/go/${item.backend}/`.

Test plan

  • `cmake --build` produces `librfdetrcpp-{avx,avx2,avx512,fallback}.so` per CPU variant
  • `nm -D librfdetrcpp-fallback.so` shows all 11 `rfdetr_capi_*` symbols exported
  • `go build -o rfdetr-cpp ./backend/go/rfdetr-cpp/` produces a 16 MB gRPC server binary
  • `go test ./core/gallery/importers/...` passes (283 specs, 0 failed, 28s)
  • End-to-end through `POST /v1/detection`: `local-ai run` + `curl` returns 9 detections for `rfdetr-base-f16` on a COCO test image, with correct `class_name` + `confidence` + `bbox` fields
  • End-to-end seg through same endpoint: 10 detections for `rfdetr-seg-nano-f16`, each with a valid PNG-encoded mask (signature `89 50 4E 47`, decoded silhouettes match the upstream PyTorch output at IoU > 0.99)

Upstream

C++ implementation at https://github.com/mudler/rf-detr.cpp (now public). The rt-detr.cpp upstream:

  • Matches PyTorch on CPU on every variant tested (1.05–1.45× speedup on Nano…Medium at T=8 on Ryzen 9 9950X3D)
  • Lossless F16 (max |Δscore| ≤ 0.006 vs F32 reference)
  • Q8_0 keeps full accuracy at 3.1× compression
  • Sub-pixel bbox match vs PyTorch (54/55 at IoU ≥ 0.95 across 7 COCO val2017 images)
  • Mask IoU 0.99+ vs PyTorch for segmentation

Assisted-by: Claude:claude-opus-4-7 [Claude Code]

@mudler mudler force-pushed the feat/rfdetr-cpp-backend branch 6 times, most recently from 0f54c59 to bdce254 Compare May 27, 2026 14:59
Adds a Go native gRPC backend that dlopens librfdetrcpp.so (built from
mudler/rf-detr.cpp at the pinned RFDETR_VERSION) via purego and exposes
the rfdetr.cpp inference pipeline through LocalAI's existing Detect RPC.

Supports all 5 RF-DETR detection variants (Nano/Small/Base/Medium/Large)
and 6 segmentation variants (SegNano/SegSmall/SegMedium/SegLarge/
SegXLarge/Seg2XLarge) with F32/F16/Q8_0/Q4_K quantizations. Pre-built
GGUFs ship at mudler/rfdetr-cpp-* on HuggingFace.

Detection returns Bbox + class_name + confidence; segmentation also
returns PNG-encoded per-detection masks via the rfdetr_capi accessor
functions (rfdetr_capi_get_detection_{class_id,box,score,class_name,
mask_png}).

End-to-end verified through POST /v1/detection: HTTP -> gRPC -> purego
dlopen -> rfdetr.cpp -> ggml -> response (9 detections on the detection
model, 21 detections + valid PNG masks on the seg-nano model against
the kitchen fixture).

Wiring:
  - backend/go/rfdetr-cpp/{main.go,gorfdetrcpp.go,CMakeLists.txt,
    Makefile,run.sh,package.sh,test.sh,.gitignore}
  - Top-level Makefile: BACKEND_RFDETR_CPP, docker-build target,
    .NOTPARALLEL, prepare-test-extra, test-extra
  - backend/go/rfdetr-cpp/Makefile: `test` target invoked by test-extra
  - .github/backend-matrix.yml: CPU + CUDA-12/13 + L4T CUDA-12/13
    (arm64) + HIP + Vulkan (amd64 + arm64) + SYCL f32/f16
  - backend/index.yaml: rfdetr-cpp meta anchor + latest/development
    image entries for every matrix tag-suffix
  - .github/workflows/bump_deps.yaml: RFDETR_VERSION pin tracking
    (mudler/rf-detr.cpp branch main)
  - gallery/index.yaml: 11 rfdetr-cpp-* entries (nano + 4 detection
    variants + 6 seg variants), all backed by mudler/rfdetr-cpp-*
    on HuggingFace with sha256 pinning on the F16 default
  - core/gallery/importers/rfdetr.go: GGUF auto-routing for HF imports
    (mudler/rfdetr-cpp-* repos route to rfdetr-cpp, Transformer-format
    repos stay on the Python rfdetr backend; explicit preferences.backend
    overrides both heuristics)
  - core/gallery/importers/rfdetr_test.go: table-driven coverage of the
    auto-routing + a live mudler/rfdetr-cpp-nano cross-check

scripts/changed-backends.js needs no change: the existing
Dockerfile.golang -> backend/go/${item.backend}/ branch already routes
the 9 rfdetr-cpp matrix entries to the correct backend path.

Assisted-by: Claude:claude-opus-4-7 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler mudler force-pushed the feat/rfdetr-cpp-backend branch from bdce254 to c30e24d Compare May 27, 2026 15:21
@mudler mudler merged commit 7a4ca8f into master May 27, 2026
63 checks passed
@mudler mudler deleted the feat/rfdetr-cpp-backend branch May 27, 2026 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants