Repository navigation
Releases
v0.3.0
TTF (two-timescale threshold flips): training ternary ({-1, 0, 1}) language models without full-precision master
weights, from scratch or from a converted ternary checkpoint.
Highlights
- Released models trained with TTF: ttf-Qwen3.8-27B-GGUF
(8.2 GB, 1.6 bits per ternary weight; GSM8K 0.895, the same as full precision) and
ttf-Qwen3.8-Flash-Next-GGUF (40.4 GB, 4.6x
smaller than the FP8 release; GSM8K 0.905, the same as FP8). Run them with
ttf-home. - The paper, Training Ternary Language Models Without Master Weights, is in
paper/(PDF + LaTeX source). - A few-minute quickstart:
recipes/demo-15m.toml(15M model, TTF vs BitNet b1.58 vs full precision).
Changes since v0.2.0
ttf.q38: Qwen3.8-Flash-Next (512-expert MoE) in TTF.- A layer-major trainer with the model and the TTF state in host memory, so one GPU trains it.
- Batched expert TTF updates and Triton grouped GEMMs.
- The PLE table from a pruned + product-quantised store.
- See
docs/Q38.md.
- The update rule:
- The accumulator leak is fixed at 1 - 1/tau, and the step-scale exponent (
--ttf-step-p) is gone. - θ follows a schedule:
--ttf-theta-start/--ttf-theta-end, defaults 4x / 5x the base, so 64 -> 16 -> 80 at
θ16. - Resuming a run with a different leak or step scale is refused.
- The accumulator leak is fixed at 1 - 1/tau, and the step-scale exponent (
- The factored update's row sketch is warm-started by default (
--dw-sketch-warm 0.75): -0.033 held-out loss at
0.8B at the same cost. The count sketch is removed. - Speed and memory:
- Head loss kernel: a Triton kernel for the frozen / trained head's CE + KD loss and gradient (bf16 logits in,
bf16 gradient out): 3.3x faster per chunk at the Qwen vocabulary, 6x less transient memory. At 0.8B conversion
recovery the step goes from 15.1 to 12.7 s on a 4070 Ti SUPER. - Micro-batch planning: the activation-checkpoint planner measures a kept block's memory directly, so batch
shapes that used to under-plan and run out of memory fit. 2 x 8 micro-batches at 0.8B run at 8.85 s/step, against
15.1 s for 1 x 16 in v0.2.0. - Orthonormalisation:
--ttf-orth auto(default) uses spanqr for
accumulator ranks of 256 and up when it is installed, else Householder QR. cholqr2 is removed (unstable in
training).
- Head loss kernel: a Triton kernel for the frozen / trained head's CE + KD loss and gradient (bf16 logits in,
update.GRAD_HOOKS: plug-ins can observe, replace or skip a layer's TTF update without patching the trainer.- Qwen3.5 / 3.8 dense models:
--center-floats(default on): the Gated DeltaNet's float gates read the centred input.--decay-ssmand--gate-lr-multoptions.- Dense exact accumulators wherever they take no more memory than the low-rank factors.
- Fixes:
- The GPU temperature check reads this process's GPU, and
--temp-check 0turns it off. --mem-snapshotdumps at any step.--mem-cap-gibuses the run's device.- The int-k embedding builds its codes in row chunks.
- The sketch basis is checkpointed.
- The GPU temperature check reads this process's GPU, and
- Profilers in
scripts/prof/(step attribution, memory, step time); move analysis inscripts/analysis/.
Needs Python 3.11+, PyTorch 2.13+, an NVIDIA GPU. Install:
pip install "ttf-ternary[data] @ git+https://github.com/oguzeldereli/ttf-ternary@v0.3.0".
License: PolyForm Noncommercial 1.0.0 (see LICENSE). Commercial use needs a separate licence from the author.
v0.2.0
TTF (two-timescale threshold flips): training ternary ({-1, 0, 1}) language models without full-precision master weights, from scratch or from a converted ternary checkpoint.
Changes since v0.1.0:
- The trainer consolidated into the
ttf/package (python -m ttf.train) with plain options and TOML recipes (recipes/scratch-110m.toml,recipes/continue-27b.toml); old command lines translate withttf.args.from_legacy(docs/OPTIONS_MAP.md). - Fused int8 ternary layers, the factored update (no dense weight gradient), the 12-bit accumulator store, host offload of the state, micro-batch accumulation into the state, pipeline (
--pipe) and data parallel training, int-k embeddings and head, distillation from an online teacher or a cached top-k. - A dense exact accumulator for full-rank layers,
--ttf-rank-ratio, rank changes on resume. - A results database (
results/runs.jsonl,scripts/results.py). - The conversion of pretrained models (GPTQ-type ternary rounding, streaming conversion) is no longer in this repository; it is a separate package.
--arch-hf(alias--init-hf) names a Hugging Face architecture whose ternary checkpoint training starts from. - Paper 1's analysis scripts:
scripts/analysis/grad_snr.py,rank_spectrum.py,rank_scaling.py.
License: PolyForm Noncommercial 1.0.0 (see LICENSE). Commercial use needs a separate licence from the author.
The TTF training and inference methods are the subject of UK patent applications GB2623290.0 and GB2624374.1 (patent pending).
v0.1.0
First archived release of the TTF (two-timescale threshold flips) code: training ternary ({-1, 0, 1}) language models without full-precision master weights, and converting pretrained models to ternary weights with the same update.
Contents:
bitnet/: trainer with packed 1.6-bit weights and low-rank optimizer state, packed-ternary Triton kernels, data-parallel training- conversion pipeline for pretrained models (ternary rounding, streaming conversion of large models)
- evaluation and inference scripts, tests
License: PolyForm Noncommercial 1.0.0 (see LICENSE). Commercial use needs a separate licence from the author.
The TTF training and inference methods are the subject of UK patent applications GB2623290.0 and GB2624374.1 (patent pending).