Skip to content

Foretoken 0.0.3

Choose a tag to compare

View all tags
@shiweijiezero shiweijiezero released this 13 Sep 22:16
v0.0.3
a2b6a78

Highlights

  • Refreshed benchmarks with conversations, recorded tool histories, synthetic output lengths, and W&B time-series and task comparisons.
  • Model sources selectable per service: Hugging Face, ModelScope, or local directories, with directory-backed runtime storage.
  • Automatic NVIDIA and MetaX runtime selection through foretoken install, with release images for both GPU families.
  • LoadBalancer discovery and optional managed address allocation during installation.
  • Expanded routing, Mooncake KV lifecycle support, and the Foretoken System Overview dashboard with alerts and diagnostics.

Thanks

  • @RSL459 for the llm-d-inspired queue-depth, KV-cache utilization, and running-request routing scorers.
  • @YehongXu1 for alert delivery integration and observability runbooks.
  • @shiweijiezero for the benchmark refactor, storage and model-source support, platform integration, and release validation.

Install

pip install --upgrade foretoken

For benchmarking:

pip install --upgrade 'foretoken[bench]'

Python 3.11 or later is required. Use the examples from this release and follow the deployment guides for platform upgrades.

Full changelog: v0.0.2...v0.0.3

Contributors

shiweijiezero, YehongXu1, and RSL459
Assets 5
Loading