Skip to content

Releases

v0.3.0

Choose a tag to compare

@oguzeldereli oguzeldereli released this 09 Nov 13:24

TTF (two-timescale threshold flips): training ternary ({-1, 0, 1}) language models without full-precision master
weights, from scratch or from a converted ternary checkpoint.

Highlights

  • Released models trained with TTF: ttf-Qwen3.8-27B-GGUF
    (8.2 GB, 1.6 bits per ternary weight; GSM8K 0.895, the same as full precision) and
    ttf-Qwen3.8-Flash-Next-GGUF (40.4 GB, 4.6x
    smaller than the FP8 release; GSM8K 0.905, the same as FP8). Run them with
    ttf-home.
  • The paper, Training Ternary Language Models Without Master Weights, is in paper/ (PDF + LaTeX source).
  • A few-minute quickstart: recipes/demo-15m.toml (15M model, TTF vs BitNet b1.58 vs full precision).

Changes since v0.2.0

  • ttf.q38: Qwen3.8-Flash-Next (512-expert MoE) in TTF.
    • A layer-major trainer with the model and the TTF state in host memory, so one GPU trains it.
    • Batched expert TTF updates and Triton grouped GEMMs.
    • The PLE table from a pruned + product-quantised store.
    • See docs/Q38.md.
  • The update rule:
    • The accumulator leak is fixed at 1 - 1/tau, and the step-scale exponent (--ttf-step-p) is gone.
    • θ follows a schedule: --ttf-theta-start / --ttf-theta-end, defaults 4x / 5x the base, so 64 -> 16 -> 80 at
      θ16.
    • Resuming a run with a different leak or step scale is refused.
  • The factored update's row sketch is warm-started by default (--dw-sketch-warm 0.75): -0.033 held-out loss at
    0.8B at the same cost. The count sketch is removed.
  • Speed and memory:
    • Head loss kernel: a Triton kernel for the frozen / trained head's CE + KD loss and gradient (bf16 logits in,
      bf16 gradient out): 3.3x faster per chunk at the Qwen vocabulary, 6x less transient memory. At 0.8B conversion
      recovery the step goes from 15.1 to 12.7 s on a 4070 Ti SUPER.
    • Micro-batch planning: the activation-checkpoint planner measures a kept block's memory directly, so batch
      shapes that used to under-plan and run out of memory fit. 2 x 8 micro-batches at 0.8B run at 8.85 s/step, against
      15.1 s for 1 x 16 in v0.2.0.
    • Orthonormalisation: --ttf-orth auto (default) uses spanqr for
      accumulator ranks of 256 and up when it is installed, else Householder QR. cholqr2 is removed (unstable in
      training).
  • update.GRAD_HOOKS: plug-ins can observe, replace or skip a layer's TTF update without patching the trainer.
  • Qwen3.5 / 3.8 dense models:
    • --center-floats (default on): the Gated DeltaNet's float gates read the centred input.
    • --decay-ssm and --gate-lr-mult options.
    • Dense exact accumulators wherever they take no more memory than the low-rank factors.
  • Fixes:
    • The GPU temperature check reads this process's GPU, and --temp-check 0 turns it off.
    • --mem-snapshot dumps at any step.
    • --mem-cap-gib uses the run's device.
    • The int-k embedding builds its codes in row chunks.
    • The sketch basis is checkpointed.
  • Profilers in scripts/prof/ (step attribution, memory, step time); move analysis in scripts/analysis/.

Needs Python 3.11+, PyTorch 2.13+, an NVIDIA GPU. Install:
pip install "ttf-ternary[data] @ git+https://github.com/oguzeldereli/ttf-ternary@v0.3.0".

License: PolyForm Noncommercial 1.0.0 (see LICENSE). Commercial use needs a separate licence from the author.

v0.2.0

Choose a tag to compare

@oguzeldereli oguzeldereli released this 28 Oct 01:30

TTF (two-timescale threshold flips): training ternary ({-1, 0, 1}) language models without full-precision master weights, from scratch or from a converted ternary checkpoint.

Changes since v0.1.0:

  • The trainer consolidated into the ttf/ package (python -m ttf.train) with plain options and TOML recipes (recipes/scratch-110m.toml, recipes/continue-27b.toml); old command lines translate with ttf.args.from_legacy (docs/OPTIONS_MAP.md).
  • Fused int8 ternary layers, the factored update (no dense weight gradient), the 12-bit accumulator store, host offload of the state, micro-batch accumulation into the state, pipeline (--pipe) and data parallel training, int-k embeddings and head, distillation from an online teacher or a cached top-k.
  • A dense exact accumulator for full-rank layers, --ttf-rank-ratio, rank changes on resume.
  • A results database (results/runs.jsonl, scripts/results.py).
  • The conversion of pretrained models (GPTQ-type ternary rounding, streaming conversion) is no longer in this repository; it is a separate package. --arch-hf (alias --init-hf) names a Hugging Face architecture whose ternary checkpoint training starts from.
  • Paper 1's analysis scripts: scripts/analysis/grad_snr.py, rank_spectrum.py, rank_scaling.py.

License: PolyForm Noncommercial 1.0.0 (see LICENSE). Commercial use needs a separate licence from the author.

The TTF training and inference methods are the subject of UK patent applications GB2623290.0 and GB2624374.1 (patent pending).

v0.1.0

Choose a tag to compare

@oguzeldereli oguzeldereli released this 15 Oct 12:03

First archived release of the TTF (two-timescale threshold flips) code: training ternary ({-1, 0, 1}) language models without full-precision master weights, and converting pretrained models to ternary weights with the same update.

Contents:

  • bitnet/: trainer with packed 1.6-bit weights and low-rank optimizer state, packed-ternary Triton kernels, data-parallel training
  • conversion pipeline for pretrained models (ternary rounding, streaming conversion of large models)
  • evaluation and inference scripts, tests

License: PolyForm Noncommercial 1.0.0 (see LICENSE). Commercial use needs a separate licence from the author.

The TTF training and inference methods are the subject of UK patent applications GB2623290.0 and GB2624374.1 (patent pending).