ThinTensor v0.2.0
ThinTensor v0.2.0 expands the native single-stream runtime and makes unsupported
Transformers causal LMs runnable through an explicit compatibility route.
Highlights:
- Native Gemma2 semantics: scaled embeddings, unit-offset RMSNorm, GeGLU,
four-norm decoder blocks, alternating local/global attention, and attention
and final-logit softcaps. - Verified Gemma-2-2B-IT max path: 57.29 tok/s versus Transformers 53.58 tok/s,
0.999555 minimum post-switch cosine, exact top-1 and ordered top-5. - Verified Qwen2.5-3B path: 93.08 tok/s versus 48.18 tok/s at 500 tokens.
- Llama candidate path: 180.28 tok/s versus 132.36 tok/s on TinyLlama-1.1B;
retained as candidate because a short BF16 cutoff tie misses strict top-5. - Full causal BF16 KV retention in every published benchmark.
- Device-budgeted guarded FP8 vocabulary heads instead of a fixed
vocabulary-size cutoff. - Phi multi-EOS generation fix, including generation_config.json EOS IDs.
- Authenticated downloads through the official
hfCLI without putting tokens
in process arguments. - Per-profile speed, peak-memory, cosine, top-k, and retention matrix in
BENCHMARKS.md. - Bundled
thintensor-corearchive binary in the Linux CPython 3.14 wheel.
The wheel SHA-256 is:
28f54aefff2b7e61d7c098cbc8e48011a6f9732a4714b45917266c1bb6ac0d5b
Performance is model-, context-, device-, driver-, clock-, and power-dependent.
max-performance remains opt-in and should be validated per model.