Skip to content

v1.4.0 — Batched NLI + ONNX Runtime

Choose a tag to compare

View all tags
@anulum anulum released this 01 Mar 04:11
· 3044 commits to main since this release
v1.4.0
7b89b8d

What's New

Batched NLI Inference (3-5x faster)

score_batch() and score_chunked() now run a single padded forward pass instead of sequential calls. Chunked document scoring is 3-5x faster.

ONNX Export + Runtime (~30-50ms/chunk GPU)

  • export_onnx() converts FactCG model to ONNX via optimum (handles DeBERTa disentangled attention)
  • NLIScorer(backend="onnx", onnx_path=...) runs inference via ONNX Runtime with auto-CUDA detection
  • New optional dep: pip install director-ai[onnx]

Other

  • ascore_batch() async helper for batched scoring
  • AggreFact benchmark predictor now batches SummaC source chunks
  • GPU device handling fix in _model_score() — inputs now move to model device

Stats

  • 680 tests passing (Python 3.10/3.11/3.12)
  • Rust crates: all green
  • Lint + type check: clean

Full Changelog: v1.3.0...v1.4.0

Assets 2
Loading