Make ONNX dependency optional

[dacorvo]

funes' embedder (bge-small-en-v1.5) and reranker (bge-reranker-base) ran exclusively on ONNX Runtime via fastembed. That one dependency decided two things: ort's prebuilt binaries require glibc ≥ 2.38, locking release binaries out of Ubuntu 22.04, and on Apple Silicon its MLAS kernels run on NEON — well below what the machine's AMX units can do.

A second inference backend

embed/rerank sit behind Embedder/Reranker traits (src/inference.rs), built through factories so no call site names a concrete ML stack. The blas backend (src/blas.rs) is a hand-written forward for both models: the encoder, tokenization, and trait impls are shared cross-platform source, and the only OS-specific code is a four-function seam — sgemm, vexp, vmax, vsum — wired to Accelerate (cblas on AMX, vForce, vDSP) on macOS and to faer's pure-Rust GEMM plus runtime-dispatched vectorized kernels on Linux. Release binaries stay baseline x86-64/aarch64 and pick AVX2/AVX-512/NEON at runtime. Weights load from safetensors, fetched into the standard HF cache on first use.

blas is the default feature; the ONNX backend remains one flag away:

cargo build --release --no-default-features --features onnx   # ONNX backend instead
cargo run --release --features onnx --example bench_backends  # A/B both backends

Building with both compiles ONNX in as the reference for bench_backends; funes itself runs blas. A build with neither backend is a compile error.

Correctness and speed

bench_backends runs two workloads — short docs (per-call overheads) and 30 docs at the 512-token cap (recall's rerank worst case, GEMM-bound):

platform	rerank	embed	agreement
macOS (Apple Silicon)	2.3× faster than ort	1.9× faster	max Δ 2.1e-5 / cosine 0.999999
Linux (64-core EPYC)	0.86× (2.55 s vs 2.19 s at 30×512)	~parity (1.16 vs 1.13 s)	max Δ 4e-6 / cosine 0.999999

On Linux the residual vs MLAS is invisible end-to-end: funes recall against a real store measures identical (~6.2 s) with either backend — recall time is search-dominated and real chunks sit well below the token cap.

Releases run on Ubuntu 22.04

Linux release builds move into an ubuntu:22.04 container: glibc compatibility is one-way, and even without ort a 24.04-built binary binds __isoc23_*@GLIBC_2.38 from the C dependencies compiled during the build. A post-build check fails the job if any symbol rises above GLIBC_2.35. Replayed locally in docker: floor 2.35, no onnxruntime in the binary, and a real-store recall runs on 22.04.

CI

Pull requests validate what default builds ship — fmt, clippy, unit and integration tests on the blas backend — and additionally clippy the onnx feature set, so a toolchain bump can't break the opt-in backend silently. Building the onnx variant and running its real-model integration test happen only on pushes to main. Model caches split accordingly: BAAI safetensors for the PR jobs, fastembed's cache for the merge-only job; the release cache keys carry the container tag so images never share compiled C objects.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR makes the ONNX/fastembed inference stack optional by introducing an Embedder/Reranker abstraction and a new default “blas” backend (Accelerate on macOS, pure Rust on Linux), while keeping an opt-in ONNX backend via a Cargo feature.

Changes:

• Introduces src/inference.rs traits + factories and migrates call sites to use them instead of fastembed types directly.
• Adds a new default blas inference backend (src/blas.rs) and makes fastembed optional behind the onnx feature.
• Updates docs/CI/release workflows to reflect backend selection and ensure build + glibc-floor constraints remain enforced.

Reviewed changes

Copilot reviewed 14 out of 15 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/scrub.rs	Switches re-embedding during scrubbing to the backend-agnostic Embedder trait.
src/recall.rs	Uses Embedder/Reranker trait objects in the cached model singleton and rerank flow.
src/lib.rs	Exposes inference module and gates blas module behind the blas feature.
src/inference.rs	Defines the backend interface and selects default backend via feature-based aliases.
src/index.rs	Makes embedding batching backend-agnostic and constructs the embedder via inference::embedder().
src/hello.rs	Uses trait-based embedding for the built-in guide dataset.
src/blas.rs	Adds the new default BLAS-based embedder/reranker implementation and model loading.
README.md	Documents default backend and how to build/run with the ONNX backend.
Cargo.toml	Makes fastembed optional; adds blas feature + optional deps and sets blas as default.
Cargo.lock	Updates lockfile for newly introduced dependencies.
build.rs	Links Accelerate only when blas is enabled on macOS.
benchmark/bench_backends.rs	Adds an example to compare backends for latency and output agreement.
AGENTS.md	Updates contributor guidance to lint both backend variants and benchmark them.
.github/workflows/release.yml	Builds inside ubuntu:22.04 container and adds a glibc symbol-floor check.
.github/workflows/ci.yml	Lints both feature variants and adds a merge-only job to integration-test the ONNX backend.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.