Skip to content

Hailo-8 encoder HEF — all-MiniLM-L6-v2 (iter 156b)

Choose a tag to compare

View all tags
@ruvnet ruvnet released this 03 May 20:35
hailo-encoder-v0.1.0-iter156b
cc490f7

First-known compiled .hef for sentence-transformers/all-MiniLM-L6-v2
on the Hailo-8 NPU (Pi 5 + AI HAT+ target).

Artifact

  • File: encoder.hef
  • Size: 15,758,361 bytes (~15 MB)
  • sha256: cdbc892765d3099f74723ee6c28ab3f0daade2358827823ba08d2969b07ebd40
  • DFC: v3.33.0
  • Hardware: hailo8

Architecture

Single-input encoder block: takes pre-computed FP32 hidden states
[1, 128, 384], returns post-encoder FP32 [1, 128, 384]. Input
embedding lookup (word + position + token_type + LayerNorm) is
handled host-side via candle's BertEmbeddings reimplementation
in crates/ruvector-hailo/src/host_embeddings.rs. Post-NPU
mean-pool + L2-normalize handled in inference.rs.

How to use

bash crates/ruvector-hailo-cluster/deploy/download-encoder-hef.sh \
    /var/lib/ruvector-hailo/models/all-minilm-l6-v2

# Then build worker with:
cargo build --release --features hailo,cpu-fallback \
    --bin ruvector-hailo-worker \
    --manifest-path crates/ruvector-hailo-cluster/Cargo.toml

Performance

Measured on cognitum-v0 (Pi 5 + AI HAT+) via cluster-bench at
concurrency=4:

Metric Value vs cpu-fallback
throughput 67.3 / sec 9.6×
p50 latency 57 ms 10×
p99 latency 152 ms 5.4×
cache hit (in-process) 15.86 M / sec 226,000×

Build provenance

Compiled iter 156b (commit ffa3e90a6 on branch hailo-backend).
The four Hailo Dataflow Compiler v3.33 SDK bugs that blocked
transformer-encoder compilation are documented in ADR-167 + ADR-175,
all worked around from user-space:

  1. KeyError on internal layer name → calibration dict keying
  2. AccelerasValueError shape mismatch → NCHW reshape
  3. ElementwiseAddDirectOp Keras deserialize → acceleras Layer
    register_keras_serializable monkey-patch (the breakthrough)
  4. tf_rgb_to_hailo_rgb align → single-input encoder form

See crates/ruvector-hailo-cluster/deploy/compile-encoder-hef.py
to rebuild from source against your own corpus.

Note on accuracy vs cpu-fallback

The single-input HEF runs full encoder attention with no padding
mask. cpu-fallback's BertModel.forward applies the real mask, so
the two embedders produce vectors in different spaces (cosine ~0.44
between matching texts). Both internally preserve semantic ordering
(sim(close) > sim(far) Δ=+0.23). The cluster's iter-143 fingerprint
separates HEF and cpu-fallback workers automatically so they never
mix in dispatch. A mask-aware HEF compile is documented as future
work in ADR-175.

Assets 3
Loading