Skip to content

v0.19.0

Latest
Latest

Choose a tag to compare

View all tags
@NanoCode012 NanoCode012 released this 10 Sep 14:19
· 2 commits to main since this release
v0.19.0
f6a1083
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Axolotl v0.19.0 Release Notes

72 commits since v0.18.0 (July 17, 2026).

We have a variety of changes this release including a new declarative model-support system, seven new model families plus ternary (BitNet b1.58) QAT, two new optimizers, a big chat-template tokenization speedup, and PyTorch 2.13 support.

Highlights

Declarative Model Support Profiles

Adding a new architecture used to mean edits scattered across the loaders. A model now picks a family template (VANILLA_CAUSAL_LM, IMAGE_TEXT_TO_TEXT) and declares only what makes it different.

Existing descriptors keep working, and there are no config changes. See the guide.

Qwen3.8-Flash-Next

You can now fine-tune Qwen's 176.94B multimodal MoE on a single B300, in as little as 120 GiB with ple_cpu_offload: true, for a minor throughput tradeoff.

QLoRA, vision QLoRA, and NVFP4 MoE-LoRA configs all ship. See the guide.

Ternary (BitNet b1.58) QAT

qat: weight_dtype: ternary trains a model with its weights restricted to {-1, 0, 1}, natively, with no extra dependency. examples/neutrino/ has the two-stage recipe Fermion used for Neutrino-1: ternary continued pretraining, then instruction tuning on top.

See the docs.

Two New Optimizers: PoLoRA and AdamC

PoLoRA, a Muon-like optimizer for LoRAs. Set optimizer: polora with LoRA or QLoRA, on single GPU, DDP, or FSDP2.

optimizer: adamc adds Adam with corrected weight decay, which scales decay by learning rate.

Chat-Template Tokenization speedup

Long multi-turn conversations no longer make preprocessing look like it has hung. An 80-turn conversation on Qwen2.5 went from 1577.6 ms to 96.4 ms, a 16.4x speedup.

The gains are most apparent with conversation length.

Six More Model Families

Ling 3.0 (Flash and Tiny), Muse Glimmer 30B, Cohere North Micro Vision Instruct, Shieldstral, Qwen3.8 27B, and Nemotron 3.5 Lightning 30B-A3B, each with example configs.

Performance & Kernel Optimizations

  • Faster bnb NF4 dequant (#3882 by @NanoCode012): pass raw pointers and ints to the bitsandbytes dequant entry points when the 0.50.0+ argtypes are present.

New Features

  • TRL 1.9.0 and new GRPO knobs (#3875 by @ved1beta): cfg.trl now surfaces entropy regularization (entropy_coef, use_adaptive_entropy, entropy_target, entropy_coef_delta/min/max), bidirectional importance-sampling clipping (vllm_importance_sampling_clip_min/max), and log_multimodal; scale_rewards widens to accept group/batch/none alongside the old boolean. See the GRPO docs.
  • Explicit flash_attention_4 and flash_attention_torch backends (#3847 by @NanoCode012): FA4 was previously reachable only by auto-upgrade from FA2; it is now a canonical attn_implementation value, alongside torch's own varlen backend, both supporting varlen sample packing. flash_attention_torch has a preflight validator that fails clearly when the installed transformers does not register it.
  • Configurable FP8 scaling recipes (#3969 by @JimmyWang0417): fp8_config.recipe selects TorchAO's tensorwise (the default), rowwise, or rowwise_with_gw_hp. fp8_enable_fsdp_float8_all_gather remains supported only with tensorwise, and incompatible rowwise combinations are rejected during validation.
  • Liger kernel backend selector (#3896 by @thad0ctor): liger_kernel_impl: cutile or cutedsl (Blackwell-tuned) opts into liger 0.8.1's alternative kernel backends, while fused linear cross-entropy stays Triton on both. Gemma 4 now routes through liger's native dispatch instead of our hand-patch, picking up the multimodal FLCE forward, and ipo under dpo_use_liger_kernel fails at config time rather than mid-run.
  • CLI help from dataclass metadata (#3855 by @vineethsaivs): argument help text is read from the help metadata key on dataclass fields, so plugin-defined args document themselves.

Documentation

  • SkyPilot guide (#3948 by @alex000kim): new SkyPilot docs covering single-node, multi-node, and spot instances, checkpoint mounting, secrets, and dataset reuse. Re-adds material dropped from the README in #2295.
  • Full attribute docstrings in the API reference (#3946 by @NanoCode012): a custom Quarto renderer so config attribute docs render in full instead of being truncated.
  • S3-compatible endpoints for dataset loading (#3978 by @ved1beta): setting AWS_ENDPOINT_URL lets you load datasets from MinIO, Tigris, Cloudflare R2, or any other S3-compatible store while keeping the s3:// path format. See the dataset loading docs.
  • delinearize-llama4 CLI invocation (#3912 by @latent-9): the documented --model / --output flags do not exist; the command takes positional arguments.
  • Broken relative links and a stale docstring (#3986 and #3985 by @simpleqt): fixes dead README and example-config links across the diffusion, llm_compressor, and nemo_gym integrations, and corrects the parameter name in the do_vllm_serve docstring.

Model & Framework Support

New Model Support

  • Qwen3.8-Flash-Next (#3965 by @NanoCode012): QLoRA, vision QLoRA, and NVFP4 MoE-LoRA configs in examples/qwen3.8-flash-next/, plus optional PLE CPU offload and CCE support.
  • Ling 3.0 Flash and Tiny (#3949 by @ved1beta): in-tree modeling copy with packing support, LoRA and QLoRA configs in examples/ling3/. Ling-3.0-tiny LoRA measures 25.0 GiB reserved at 12.3 s/step on one H100 80GB.
  • Muse Glimmer 30B (#3931 by @NanoCode012): multimodal, with examples/muse-glimmer/ QLoRA text and vision configs and Liger support.
  • Cohere North Micro Vision Instruct (#3932 by @NanoCode012): a cohere_compass multimodal profile with image-text processing, plus QLoRA and FFT examples.
  • Shieldstral (#3922 by @NanoCode012): text and vision LoRA configs, Liger for Mistral3/Ministral3, improved Mistral3 role masking and tokenizer discovery.
  • Qwen3.8 27B (#3935 by @ved1beta): QLoRA example config, along with sample-packing fixes across the Qwen3.5 base/text/MoE variants.
  • Nemotron 3.5 Lightning 30B-A3B (#3945 by @ved1beta): a QLoRA config for the new checkpoint, with the Nemotron-H modeling patch rewritten against upstream and the stale gradient-checkpointing restriction removed.

Dependency Updates

Bug Fixes

  • orig_mod prefix in FSDP2 + compile checkpoints (#3958 by @NanoCode012): drop the now-redundant accelerate patch that upstream fixed, which was leaking _orig_mod into saved state dicts, and stop PhiForCausalLM being mangled to hiForCausalLM by an lstrip("FSDP") that strips characters rather than a prefix. Saved tensors verified bit-identical. Also rejects cpu_ram_efficient_loading with load_in_4bit on FSDP2, where the rank-0-only quantization deadlocks the state-dict scatter.
  • Fully-masked samples were never dropped (#3864 by @vineethsaivs): drop_no_trainable_tokens compared a Python list to -100 under np.any, which always evaluates truthy, so samples with zero trainable tokens survived dataset preparation. The comparison is now element-wise, and a dataset left empty by the filter raises with a pointer at train_on_inputs / roles_to_train / the chat template.
  • Silent out-of-bounds writes in MoE expert kernels (#3865 by @winglian): expert weight stacks over 2^31 elements (512x5120x2048 at Nemotron-3-Ultra scale) overflowed the i32 expert-index times stride products, corrupting neighboring allocations in the grouped dW kernel. Expert indices and row ids are promoted to i64 at every stride multiplication, across both the ScatterMoE LoRA kernels and the SonicMoE NVFP4 Triton kernels.
  • NF4 dequant queued work on a stale CUDA stream (#3888 by @NanoCode012): the fast dequant path cached current_stream() per device, but the current stream belongs to the caller's context, so a caller running under a non-default stream had its dequant queued on the wrong one. The stream is now resolved per call, read from torch rather than a bitsandbytes internal that upstream may remove.
  • SSM state leaking across packed sequences on Nemotron (#3945 by @ved1beta): a Mamba2 chunk scan that silently ignores seq_idx mixes state between packed samples. The resolved scan is now probed once at startup and raises with an install hint instead of training quietly wrong.
  • Qwen3.5 / Qwen3-Next patches broken by transformers 5.15 (#3937 by @NanoCode012): self.layer_type was renamed and fla moved to module level, silently breaking the optimized attention patches. Now covered by tests.
  • FA4 auto-upgrade silently broke context parallelism (#3928 by @thad0ctor): a configured flash_attention_2/flash_attention_3 was upgraded to FA4 whenever the host supported it, but ring attention only substitutes the flash_attention_2 dispatch key, so each rank attended over its own shard alone, producing wrong loss and gradients with no error. The implicit upgrade is now skipped when context_parallel_size > 1; an explicitly requested flash_attention_4 is unaffected.
  • LoRA shards left untrained under FSDP2 cpu_ram_efficient_loading (#3977 by @winglian): every LoRA shard now trains, and the process group is initialized before load for expert parallelism (fixes #3772, which #3779 did not fully address).
  • Qwen3.5 sample packing crashed in get_cu_seqlens (#3975 by @Anai-Guo): packed position_ids arrive as a non-contiguous sliced view that view(-1) rejects; reshape(-1) is used instead, leaving the already-contiguous path bit-for-bit unchanged (fixes #3973).
  • FA2 hub kernel pinned to v2 (#3968 by @NanoCode012): the v3 entry in transformers' flash-attention kernel mapping breaks the GQA backward, so the FA2 path resolves the v2 kernel explicitly. The FA2 availability check also stops reporting valid hub-kernel setups as unavailable.
  • Kimi Delta Attention gating (#3949 by @ved1beta): align the fused_kda_gate call with the signature fla changed in 0.4.1, and extract the remote-code redirect into a shared helper that Ling 3.0 reuses, cutting patch_kimi_linear.py from 95 lines to 15.
  • SonicMoE against the upstream kernel (#3954 by @NanoCode012): call upstream's public expert forward instead of re-creating the shim it removed, so an upstream attribute change raises rather than silently mis-training. Also respects use_kernels: false and errors on unsupported models. Credit to @OnePunchMonk for the initial report.
  • FA4 under quack 0.6 and latest transformers (#3847 by @NanoCode012): fixes the FA4 path against the quack 0.6 / cutlass-dsl 4.6.0 stack and current transformers. FA4 now requires quack-kernels>=0.6.0 on nvidia-cutlass-dsl==4.6.0; older quack raises cudaErrorIllegalInstruction in the backward, and on Blackwell the FA4 backward hangs without a patched build. See the attention docs for install steps.
  • Merged builtin checkpoint conversions (#3893 by @NanoCode012): register weight conversions read-merge-write so a profile's conversions no longer replace the built-ins transformers ships. The NVFP4 expert converters still register in replace mode, and stacking the two under one model_type now raises instead of silently dropping one.
  • Gemma merge produced gibberish (#3758 by @ved1beta): re-sharding a merged model wrote the tensor count into the index's total_size field instead of the byte size; the index now reports both total parameters and total size.
  • fp32/CPU tensors routed into the varlen flash kernel (#3885 by @winglian): the sdpa_varlen fast path guarded on mask, dropout, head_dim, and scaling but not dtype or device, so sdpa + sample_packing in fp32 crashed on torch 2.12.1. Such rows fall back to stock SDPA with the rebuilt block-diagonal mask, keeping documents isolated.
  • Selective activation checkpointing under saved-tensor hooks (#3942 by @Excelius-Wang): feature-detect PyTorch's respect_saved_tensors_hooks argument and set it explicitly, avoiding the staged warn-then-error behavior and double-processing of SAC-cached tensors (fixes #3883).
  • ddp_timeout ignored during weight loading (#3741 by @ved1beta): ddp_timeout only reached TrainingArguments, too late, since init_distributed_state() has already built the process group on the default 1800s NCCL timeout. Rank 0 loading large weights for over 30 minutes still tripped it. prepare_optim_env now sets AXOLOTL_NCCL_TIMEOUT before any distributed init, so FSDP, DeepSpeed, and DDP all honor it (fixes #3443).
  • Packing stats when drop_last discards a batch (#3849 by @Anai-Guo), micro_batch_size / gradient_accumulation_steps defaults ignored in check_batch_size_fields (#3837 by @Anai-Guo).
  • trust_remote_code handling (#3858 by @NanoCode012): respected properly during packing; vLLM serve and merge-lora now forward model revision and trust_remote_code (#3960 by @dudeperf3ct).
  • EBFT config validator bypassed (#3887 by @NanoCode012), KTO argilla-chat follow-ups (#3886 by @NanoCode012).
  • Mistral tokenizer cache_dir (#3859 by @thad0ctor): stop passing the literal string "None" as the cache directory.
  • Gemma 4 vision batches crashed under FA2 (#3880 by @ved1beta): the hybrid-mask patch forced SDPA on sliding-mask builds too, handing the FA2 sliding layers a 4D mask and tripping a device-side assert in _upad_input. Calls carrying and_mask_function, which is every Gemma 4 batch that goes through create_masks_for_vision_model, now pass through untouched.
  • Gemma 4 31B LoRA gradient-norm spikes (#3987 by @NanoCode012): Gemma 4's text attention uses scaling=1.0 with QK-RMSNorm, leaving the softmax structurally near-saturated, so LoRA runs hit intermittent pre-clip gradient norms 100-1000x their neighbours with no matching loss spike (transformers#45676). The 31B configs compensate with learning_rate: 5e-5 and max_grad_norm: 0.1, and a 32k-sequence 31b-lora-fsdp.yaml is added.
  • Stale numpy slicing TODOs removed (#3921 by @n-dlms).

Infrastructure

  • PyTorch 2.13 images (#3877 by @winglian): latest GPU containers move from 2.12.0 to 2.12.1 and gain 2.13.0 variants, nightlies follow to 2.12.1, the stale 2.12.0 base image is dropped, and two 2.13.0 docker e2e jobs are added.
  • Docs build fixed (#3924 by @NanoCode012); Mistral tokenizer fixtures load offline (#3970 by @NanoCode012) so CI stops tripping HF rate limits.
  • Nightly tests fixed (#3980 by @NanoCode012): bump fla to 0.5.2 after transformers began nested-importing it, install kernels from git main in the nightly workflow, and restore the patched PEFT _inject_parameters contract for target parameters.
  • Dependabot Action bumps: setup-uv 8.2.0 → 10.0.1 (#3939), docker/login-action 3.7.0 → 4.6.0 (#3871, #3927), pypa/gh-action-pypi-publish 1.14.0 → 1.14.2 (#3913, #3925), actions/checkout 7.0.0 → 7.0.1 (#3872), actions/github-script 7.1.0 → 9.0.0 (#3870), actions/setup-python 5.6.0 → 7.0.0 (#3850, #3914), docker/build-push-action 5.4.0 → 7.3.0 (#3852), docker/metadata-action 5.10.0 → 6.2.0 (#3851), docker/setup-buildx-action 4.2.0 → 4.3.0 (#3966), peter-evans/create-pull-request 6.1.0 → 8.1.1 (#3854), marocchino/sticky-pull-request-comment (#3853).

New Contributors

Full Changelog: v0.18.0...v0.19.0

Contributors

winglian, thad0ctor, and 12 other contributors
Assets 2
Loading