feat(backend): rfdetr-cpp native object detection + segmentation backend#10028
Merged
Conversation
0f54c59 to
bdce254
Compare
Adds a Go native gRPC backend that dlopens librfdetrcpp.so (built from
mudler/rf-detr.cpp at the pinned RFDETR_VERSION) via purego and exposes
the rfdetr.cpp inference pipeline through LocalAI's existing Detect RPC.
Supports all 5 RF-DETR detection variants (Nano/Small/Base/Medium/Large)
and 6 segmentation variants (SegNano/SegSmall/SegMedium/SegLarge/
SegXLarge/Seg2XLarge) with F32/F16/Q8_0/Q4_K quantizations. Pre-built
GGUFs ship at mudler/rfdetr-cpp-* on HuggingFace.
Detection returns Bbox + class_name + confidence; segmentation also
returns PNG-encoded per-detection masks via the rfdetr_capi accessor
functions (rfdetr_capi_get_detection_{class_id,box,score,class_name,
mask_png}).
End-to-end verified through POST /v1/detection: HTTP -> gRPC -> purego
dlopen -> rfdetr.cpp -> ggml -> response (9 detections on the detection
model, 21 detections + valid PNG masks on the seg-nano model against
the kitchen fixture).
Wiring:
- backend/go/rfdetr-cpp/{main.go,gorfdetrcpp.go,CMakeLists.txt,
Makefile,run.sh,package.sh,test.sh,.gitignore}
- Top-level Makefile: BACKEND_RFDETR_CPP, docker-build target,
.NOTPARALLEL, prepare-test-extra, test-extra
- backend/go/rfdetr-cpp/Makefile: `test` target invoked by test-extra
- .github/backend-matrix.yml: CPU + CUDA-12/13 + L4T CUDA-12/13
(arm64) + HIP + Vulkan (amd64 + arm64) + SYCL f32/f16
- backend/index.yaml: rfdetr-cpp meta anchor + latest/development
image entries for every matrix tag-suffix
- .github/workflows/bump_deps.yaml: RFDETR_VERSION pin tracking
(mudler/rf-detr.cpp branch main)
- gallery/index.yaml: 11 rfdetr-cpp-* entries (nano + 4 detection
variants + 6 seg variants), all backed by mudler/rfdetr-cpp-*
on HuggingFace with sha256 pinning on the F16 default
- core/gallery/importers/rfdetr.go: GGUF auto-routing for HF imports
(mudler/rfdetr-cpp-* repos route to rfdetr-cpp, Transformer-format
repos stay on the Python rfdetr backend; explicit preferences.backend
overrides both heuristics)
- core/gallery/importers/rfdetr_test.go: table-driven coverage of the
auto-routing + a live mudler/rfdetr-cpp-nano cross-check
scripts/changed-backends.js needs no change: the existing
Dockerfile.golang -> backend/go/${item.backend}/ branch already routes
the 9 rfdetr-cpp matrix entries to the correct backend path.
Assisted-by: Claude:claude-opus-4-7 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
bdce254 to
c30e24d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a Go native gRPC backend (
backend/go/rfdetr-cpp/) that dlopenslibrfdetr.so(built from mudler/rf-detr.cpp at the pinnedRFDETR_VERSION) via purego and exposes the rfdetr.cpp inference pipeline through LocalAI's existingDetectRPC.Supports all 5 RF-DETR detection variants (Nano/Small/Base/Medium/Large) and 3 segmentation variants (SegNano/SegSmall/SegMedium) with F32/F16/Q8_0/Q4_K quantizations. Pre-built GGUFs ship at
mudler/rfdetr-cpp-*on HuggingFace (32 models total).Detection returns Bbox + class_name + confidence; segmentation also returns PNG-encoded per-detection masks via the
rfdetr_capi_get_detection_{class_id,box,score,class_name,mask_png}accessor functions.Wiring (full adding-backends.md checklist)
backend/go/rfdetr-cpp/{main.go, gorfdetrcpp.go, CMakeLists.txt, Makefile, run.sh, package.sh, test.sh, .gitignore}Makefile:BACKEND_RFDETR_CPP,docker-build-rfdetr-cpp,.NOTPARALLEL,prepare-test-extra,test-extra.github/backend-matrix.yml: CPU + CUDA 12 + CUDA 13 + HIP + Vulkan + SYCL f16/f32 + L4T variantsbackend/index.yaml:&rfdetrcppmeta + 17 image entries (latest + development for each variant).github/workflows/bump_deps.yaml:RFDETR_VERSIONpin tracking againstmudler/rf-detr.cppmain branchgallery/index.yaml:rfdetr-cpp-nano+rfdetr-cpp-basemodel entriescore/gallery/importers/rfdetr.go: GGUF auto-routing for HF imports (routes `.gguf`-bearing repos to `rfdetr-cpp`, falls through to Python `rfdetr` backend otherwise)core/gallery/importers/rfdetr_test.go: 6-entry table-driven test + live HF cross-check against `mudler/rfdetr-cpp-nano`The `scripts/changed-backends.js` registration was checked and is not needed — the existing `endsWith("golang")` branch already routes `./backend/Dockerfile.golang` entries to `backend/go/${item.backend}/`.
Test plan
Upstream
C++ implementation at https://github.com/mudler/rf-detr.cpp (now public). The rt-detr.cpp upstream:
Assisted-by: Claude:claude-opus-4-7 [Claude Code]