🚀 ThinTensor Release v1.0.1
📦 Optimized Architectures & Speedups
- Gemma-4-E2B: Achieved 43.60 tok/s (vs 1.62 tok/s HuggingFace baseline, 26.91x speedup).
- OLMoE-1B-7B: Achieved 33.70 tok/s (vs 9.22 tok/s HuggingFace baseline, 3.65x speedup).
- Both models achieve 0.999+ cosine similarity correctness.
💡 Documentation Note: Qwen-0.8B Sample Behavior
If you run the sample Qwen3.5-0.8B model, note that:
- Document Completion: It is a raw base model (not chat/instruct), so it performs raw next-token document completion. It naturally prepends commas or continuations (e.g.,
Hello->, I am working with...). - Greedy decoding repetitions: Under greedy decoding (required for correctness/parity checking), tiny models (0.8B) lack the stochastic sampling needed to break out of repetition loops.
- For interactive conversational chat, download and convert fine-tuned instruct models (e.g.
Qwen/Qwen2.5-3B-Instruct).