v1.0.0

[HenryNdubuaku]

Performance

• Industry-leading speed: Up to 75 tokens/second on modern phones for Qwen3-INT8
• Broad device support: Runs efficiently on devices from iPhone X to the latest flagships
• Compact model format: 394MB for Qwen3-600M-INT8 (2x smaller than alternatives)

Battery Efficiency

• Ultra-low power consumption: 0.33-0.50% battery drain per minute
• CPU-optimized: Avoids GPU thermal throttling and excessive heat generation
• Sustainable inference: Run models for hours without significant battery impact

Features

Core Engine

• Multi-model support: Qwen, Gemma, SmolLM, LFM2, and Nomic models
• Quantization: INT8 precision with INT4 coming soon
• Context management: Sliding window attention with configurable KV-cache
• Tool calling: Native support for function calling in compatible models
• OpenAI-compatible API: Drop-in replacement for existing implementations

Graph Framework

• Zero-copy operations: Memory-efficient tensor computations
• Mixed precision: Automatic precision management (INT8/FP16/FP32)
• Memory-mapped weights: Instant model loading without RAM duplication
• Custom operations: Full suite of transformer primitives (RoPE, RMSNorm, etc.)

ARM Optimizations

• NEON SIMD: Vectorized operations for all major kernels
• ARMv8.6 i8mm: Hardware-accelerated INT8 matrix multiplication
• Cache-friendly: Optimized memory access patterns
• Platform-specific tuning: Adaptive thresholds for different devices

Supported Models

This release includes support for:

• Language Models: 270M to 1.7B parameters
  • Google Gemma-3 (270M, 1B)
  • Qwen3 (600M, 1.7B)
  • SmolLM2 (360M)
  • LiquidAI LFM2 (350M, 700M, 1.2B)
• Embedding Models:
  • Qwen3-Embedding (600M)
  • Nomic-Embed-Text-v2-MoE

Community

• Documentation
• Discord
• GitHub Issues

---

This discussion was created from the release v1.0.0.