v0.4.0 — Metal GPU Dispatch + BPE Tokenizer + Streaming

What's New

🎮 Real Metal GPU Dispatch (metal-rs)

• Full GPU compute pipelines for matmul, RMSNorm, softmax, RoPE, SiLU
• Automatic CPU/GPU dispatch based on tensor size (>4096 elements → GPU)
• No more shelling out to xcrun — direct Metal API via metal-rs crate

🔤 BPE Tokenizer

• Standard BPE with merge rules from GGUF metadata
• SentencePiece-style BPE using token scores
• Byte-level fallback for unknown characters
• Single-token decode for streaming output

📡 Streaming Responses

• Ollama endpoints: chunked transfer encoding (NDJSON), token-by-token
• OpenAI endpoint: Server-Sent Events (SSE)
• stream parameter support (default: true for Ollama, false for OpenAI)

Full Changelog

See CHANGELOG.md for details.

Roadmap

• v0.1 — GGUF parser, mmap loader, LRU cache, prefetcher, CPU inference
• v0.2 — Metal compute foundation, SIMD ops, API server
• v0.3 — KV cache, Metal shaders, SwiGLU FFN, quantized kernels
• v0.4 — Metal GPU dispatch, BPE tokenizer, streaming responses ← you are here
• v0.5 — Speculative decoding with draft model
• v1.0 — Production-ready, benchmarked against llama.cpp