Skip to content

6.3.0

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 18 Aug 12:42
v6.3.0
aabe6ce

✨ New Features

LLM

  • SafeTensors Support: Added support for Hugging Face's SafeTensors format, including write functionality for efficient and secure tensor serialization.
  • KV Caching: Memory-efficient key-value cache using a shared Radix-backed pool for faster LLM inference
  • FlashAttention: Leverages FlashAttention for optimized attention computation

Serve - API

  • Unified Model Catalog: All models (Chat, SMILE, ONNX) now listed via a single GET /api/v1/models endpoint with OpenAI-compatible format
  • Conversation Management: Full CRUD operations for chat conversations via the new Conversation resource
  • Streaming & Non-Streaming: Chat completions support both streaming (SSE) and non-streaming JSON responses

Serve - Inference UI

  • Unified Inference Page: New /infer page combines Chat, SMILE, and ONNX inference in one interface
  • Vision Mode: ONNX inference now supports image inputs with automatic preprocessing (resize, normalization)
  • Virtualized Results: Large batch results display efficiently with virtual scrolling
  • Split Layout: Form and results display side-by-side for better workflow

Docker & Deployment

  • GPU Docker Image: Multi-stage build with CUDA 13.2 support and LibTorch
  • CPU Docker Images: Upgrade to UBI 10 with OpenJDK 25 runtime

🔧 Improvements

Performance

  • Faster File Sync: SafeTensors loading now uses positioned FileChannel I/O instead of memory-mapping (handles multi-GB checkpoints)
  • Optimized Memory: ONNX Runtime memory footprint restricted; GPU memory fraction configurable

Developer Experience

  • Better Error Messages: ONNX runtime loading failures now show clear setup hints
  • Tokenizer Auto-Download: HuggingFace tokenizers downloaded automatically when missing
  • Model Architecture Info: GET /api/v1/models/{id} now returns detailed model metadata (architecture, shapes, etc.)

🐛 Bug Fixes

Core

  • Big-Endian Float Corruption: Fixed ONNX tensor byte order handling (was causing scrambled predictions on big-endian systems)
  • SSE Race Conditions: Chat streaming now subscribes before generation starts, preventing lost chunks

Testing & Reliability

  • Test Port Conflicts: Fixed flaky tests caused by port 8081 being in use (now picks random free port)
  • Association Rules Tests: Fixed race conditions in FP-Growth tests causing flaky counts

HuggingFace Integration

  • Broken Symlinks: Fixed tokenizer resolution when snapshot links were incorrectly computed
  • Model Validation: Chat service now fails cleanly with 503 when model is unavailable

Studio

  • Code Cells: add null safety and fix evalMagic in code cells by @TsybulkaM in #884
  • Theme Setting: Fixed theme setting logic in SettingsDialog by @TsybulkaM in #885

🔒 Security

  • Native Library Loading: Improved error handling for CUDA and ONNX native library initialization

⚠️ Breaking Changes

  • Model List Endpoints: GET /api/v1/ml/models and GET /api/v1/onnx removed; use GET /api/v1/models instead
  • Scala VegaLite API: Moved from smile.plot.vega to smile.vega (updated imports required)

📦 Dependencies

  • Torch: Upgraded to 2.13
  • ONNX Runtime: Upgraded to 1.29
  • SBT: Upgraded to 2.0.6
  • CUDA: Support for CUDA 13.2

Full Changelog: v6.2.5...v6.3.0

Contributors

TsybulkaM
Assets 3
Loading