Skip to content

Chat Template API, 1024 Context Window, & Critical GPU Fixes

Pre-release
Pre-release

Choose a tag to compare

View all tags
@qwatts-dev qwatts-dev released this 23 Feb 00:39
v0.12.0
29bc9ea
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

This release transforms bitnet.js from a raw inference engine into a developer-friendly library, introducing a proper Chat API, expanding the context window, and fixing a critical GPU math bug that caused generation collapse.

New Features

  • Chat Template API: Added engine.chat() method with native support for Llama 3 chat formatting (<|start_header_id|>, <|eot_id|>, etc.).
  • Expanded Vocabulary: Updated Python extraction scripts to include 10 Llama 3 special tokens (vocab size increased from 16,385 to 16,395).
  • Expanded Context Window: Increased MAX_SEQ_LEN from 128 to 1024 tokens, allowing for much longer multi-turn conversations.
  • Sampling Parameters: Wired temperature, topK, topP, and repetitionPenalty through the .chat() options (previously hardcoded).
  • Examples Directory: Added a new examples/chat/ directory featuring a fully interactive, streaming multi-turn chat UI.

Bug Fixes

  • CRITICAL: Strided Softmax Bug: Fixed a severe WGSL shader bug in SHADER_GQA_ATTENTION where tokens beyond index 127 were never normalized due to a flat workgroup size mismatch. The softmax phases (max, exp+sum, normalize) now correctly use strided loops.
  • Streaming UI Fix: Fixed a DOM firstChild null reference error in the chat example that occurred during token streaming.

Structural Changes

  • Landing Page: Completely redesigned the root index.html into a proper library landing page with Quick Start, API docs, and example cards (using Feather Icons).
  • Test Suite Relocation: Moved the automated GPU validation tests from the root to a dedicated tests/ directory.
Assets 2
Loading