TensorLib v0.3.0

[0.3.0] - 2026-09-21

TensorLib 0.3.0 improves CPU training performance and introduces CommandLM as
an experimental end-to-end language-model example. CommandLM demonstrates the
pipeline and safety boundaries; its generated-command quality is not yet
production-ready.

Training and autograd performance

• Added leaf-only gradient retention so training can discard intermediate
  gradients while preserving the existing retain-all behavior by default.
• Fused sibling slice-gradient accumulation and added fast paths for leading
  and middle-axis reductions, same-shape contributions, and persistent
  gradient merges.
• Parallelized large autograd fills, contiguous gradient reductions,
  million-element tensor operations, row-contiguous and transposed clones,
  large right-hand-side packing, and AdamW validation and gradient norms.
• Improved packed-matrix scheduling by splitting columns adaptively and reduced
  avoidable storage-sized leaf-gradient copies.

On the documented i5-13400F benchmark host at 16 threads, these changes reduced
the approximately 30-million-parameter CommandLM training step by 29.1% at
batch 1 and 20.4% at batch 16. AdamW time fell from about 74.5 ms to 36.1 ms.
These results are workload- and machine-specific; the full methodology and
phase breakdown are in docs/commandlm_cpu_optimization.md.

Experimental CommandLM example

• Added a configurable 29.6-million-parameter structured-command model with
  training, validation, inference, explicit resume support, and atomic
  checkpoints containing model, AdamW, and RNG state.
• Added a deterministic 1,024-token BPE-style tokenizer with byte fallback,
  versioned vocabulary persistence, and round-trip tests.
• Added deterministic corpus generation plus import and audit tooling for the
  NL2Bash and CLI-1M datasets.
• Added conservative output validation: the example emits allowlisted command
  records, rejects shell metacharacters and dangerous commands, and never
  executes generated text.
• Added CommandLM-specific benchmark shapes and labeled training-phase metrics.

CommandLM is released as an experimental example. Its current model outputs may
be poor without a carefully curated corpus, sufficient training, and separate
quality evaluation, so generated commands should be inspected rather than
trusted or executed.

Quality

• Expanded tests for leaf-only gradient retention, fused slice gradients,
  tensor layouts, parallel AdamW validation and rollback, and tokenizer
  persistence.
• Hardened corpus parsing, batch validation, checkpoint compatibility checks,
  and explicit checkpoint resume behavior.

TensorLib v0.2.0

TensorLib 0.2.0 is a performance-focused release that substantially reduces
TinyLM forward and training costs while preserving deterministic gradients,
checkpoint compatibility, and the public tensor API.

Optimization highlights

• Parallel contiguous, packed, and batched matrix multiplication with adaptive
  OpenMP scheduling.
• Cached and reusable transpose-aware matrix packs across Linear forward and
  backward passes.
• Direct batched RHS-gradient and projected weight-gradient kernels, including
  deterministic vectorized Linear weight-gradient GEMM.
• Fast paths for contiguous cloning, reductions, broadcast operations, and
  slice backward; tiled strided cloning and stride-cursor broadcast iteration.
• Fused LayerNorm, softmax/log-softmax, indexed cross-entropy, and GELU backward
  kernels, with deterministic parallel backward paths.
• Lower-overhead autograd through first-gradient ownership transfer, in-place
  compatible accumulation, producer-side broadcast reduction, and reusable
  traversal storage.
• Faster AdamW through contiguous single-pass updates, specialized gradient
  scanning, and deterministic parallel updates.

On the repository's i5-13400F reference host, the matched full-profile
benchmarks measured gains of 1,186% for LayerNorm, 1,099% for transposed copy,
261% for causal attention, 164% for TinyLM forward, and 141% for a decoder
block. Results are machine- and workload-specific; see CHANGELOG.md and the
benchmark documentation for methodology and the complete breakdown.

TinyLM checkpoint

This release includes a new tiny_lm.chk for the fixed 1,902,976-parameter
TinyLM preset (4 layers, width 192, 6 heads, context 128, byte vocabulary 256).
It includes model weights, AdamW state, and RNG state and can be resumed with:

./build/tiny_lm CORPUS --checkpoint tiny_lm.chk --resume

SHA-256:
bc11a667129224f64f9ede0788bb6367206dc912ac1e42a5bca5c5115edd21fc

Verification

• 37/37 tests passed in a Release build with warnings treated as errors.
• The release checkpoint loaded successfully and produced validation loss
  1.27012 against the local evaluation corpus.
• All local Markdown links passed validation.

Full changelog: https://github.com/nisbenz/TensorLib/blob/v0.2.0/CHANGELOG.md

v0.1.0

Changelog

[0.1.0] — 2026-07-27

Initial release of TensorLib — a from-scratch deep learning tensor library written entirely in C with zero external ML dependencies.

Added

• Tensor Core — N-dimensional float32 arrays with NumPy-style broadcasting, zero-copy strided views, reshape, transpose, slice, squeeze, expand, and reference-counted storage with mutation version tracking.
• Tensor Operations — Element-wise arithmetic (add, sub, mul, div, pow, neg, exp, log, sqrt), activations (ReLU, sigmoid, tanh, GELU), reductions (sum, mean, max), gather, and generalized strided matmul.
• AVX2+FMA Matmul — Blocked micro-kernel (MR=4, NR=16) with packed-RHS optimization and batched strided dispatch. Tile sizes MC=64, NC=64, KC=128. Benchmarked against OpenBLAS.
• Autograd Engine — Dynamic reverse-mode automatic differentiation over 23 differentiable operations. Stale-graph detection via storage version counters. Transactional gradient accumulation with topological traversal.
• Neural Network Modules — Composable C-style OOP module hierarchy with function-pointer dispatch:
  • Linear (with bias), Embedding, learned Positional Embedding, LayerNorm, Dropout
  • Multi-Head Causal Self-Attention (fused QKV projection, causal masking)
  • Pre-Norm Decoder Block, full GPT-style Decoder stack
• Loss Functions — Cross-entropy (numerically stable), Softmax, LogSoftmax.
• Optimizers — SGD with momentum, AdamW with decoupled weight decay, bias correction, and gradient clipping.
• Checkpointing — Versioned, atomic (transaction-safe) binary save/load of model parameters, optimizer state, and RNG state.
• Deterministic RNG — Splitmix64 PRNG with uniform and normal (Box-Muller) distributions.
• Examples:
  • TinyLM — Byte-level decoder language model (~1.9M params, 4-layer, 192-width, 6-head, 128-context) trained with AdamW on raw byte corpora. Includes checkpoint/resume and text generation.
  • MNIST MLP — 784→128 ReLU→10 Softmax classifier trained with SGD and cross-entropy.
  • Autograd Example — Computation graph visualization demo.
• Test Suite — 33 unit test executables covering tensor ops, autograd, all NN modules, optimizers, and checkpointing.
• CI — GitHub Actions workflow running full test suite on Alpine Linux.

Build System

• CMake and GNU Makefile with -O3 -march=native -flto -fopenmp optimizations.
• OpenMP-accelerated matmul and element-wise kernels.