Skip to content

v0.6.0

Choose a tag to compare

View all tags
@SStas SStas released this 04 Apr 16:02
· 7 commits to main since this release
v0.6.0
394aac1

v0.6.0 — Public Connector API

Highlights

  • Public connector API — hidden_dim, num_layers, context_length, vocab_size, device, dtype, tokenize(), detokenize(), apply_chat_template(), stop_token_ids, stop_strings on the EngineConnector ABC
  • OutputType/PayloadType split — API-level enum (OutputType) separated from wire-level enum (PayloadType). PayloadType.AUTO removed.
  • LlamaCpp latent primitives — create_inference_context(), run_latent_steps(), keep_context=True, grammar= for constrained generation
  • 11 bug fixes including critical HF inject_and_generate() crash and LlamaCpp use-after-free

Breaking changes

  • tokenize() returns List[int] (was Any/torch.Tensor)
  • PayloadType.AUTO removed — use OutputType.AUTO
  • extract_hidden_state()/inject_and_generate() removed from ABC
  • LlamaCpp tokenize() no longer adds BOS

See CHANGELOG.md for full details.

Assets 2
Loading