xLLM 1.1.1 Release

The release of xLLM version 1.1.0, with the following updates:

Highlights

• Add logging features for MoE and MoVA blocks

xLLM 1.1.0 Release

The release of xLLM version 1.1.0, with the following highlights:

Highlights

Add sampled health metrics for training analysis

Opt-in, sampled training-health logging: per-layer activation statistics, Q/K statistics,
per-matrix weight and gradient norms, logit statistics and Gekko branch/memory statistics,
written to a separate health logger. Off by default; no cost when off and none on
unsampled steps. Ports xllm-lab/xllm #209, #210, #212, #213 and the Gekko parameter norms
from #280, and adds the Gekko core statistics, a logit-stats fix for Gekko's fused output
layer, and a skip-with-warning policy for incomplete steps.

Usage

--health_metrics true --health_log_freq 100 \
--health_query_key_stats true --health_weight_norms true --health_gradient_norms true \
--health_logits_stats true --health_gekko_stats true

Gating: health_metrics requires arch ∈ {transformer, gekko}; Gekko additionally requires
softmax attention and dense layers; matrix norms require dense layers and SwiGLU;
health_gekko_stats requires Gekko and implies Q/K statistics. Output goes to
metrics.health.jsonl and to TensorBoard/W&B under the unprefixed metric names.

xLLM 1.0.2 Release

The release of xLLM version 1.0.1, with the following updates:

Highlights

• Fix bugs in partial rope dim in fused block mode
• Use FP32 normalized weight in causal conv
• Add an arg to manually set wandb name

xLLM 1.0.1 Release

xLLM 1.0.1

The release of xLLM version 1.0.1, with the following updates:

Highlights

• Add xattn backend for flash causal attention.
• Remove loss in the prediction of the first tokens.
• Refactor qk-norm in MHA and GDA.
• Add attn_stability_control arg.

Fixes

• Fix bug in MoVA fused block function.
• Fix bug in multi-head attention with swift backend

xLLM 1.0.0 Release

The first release of xLLM (version 1.0.0)