Conversation
Revert "FSDP2+LoRA integration fixes"
Revert "Add RL recipe for Math/GSM8K training"
* Selective gradient checkpointing with moe_act support Per-submodule selective checkpointing and moe_act activation recompute for MoE models with Expert Parallelism. Avoids expensive EP all-to-all recomputation during backward while maintaining memory efficiency. ## Selective Checkpointing (Phase 1) - Add `recompute_modules` and `moe_checkpoint_method` to TrainingArguments - Per-submodule checkpoint in Qwen3MoeDecoderLayer: attention and MLP can be independently checkpointed via `recompute_modules: ["self_attn"]` - Conditional routing replay: disabled when MoE isn't recomputed - Config propagation through torch_parallelize → base model → decoder layers ## moe_act Autograd Functions (Phase 2) - Triton: TritonEPGroupGemmMoeAct, TritonMoeExpertsFunctionMoeAct - Quack: QuackEPGroupGemmMoeAct, QuackMoeExpertsFunctionMoeAct - Native: checkpoint-based gate+up recompute via torch.utils.checkpoint - All variants save 5 tensors (EP) / 8 tensors (local) instead of 7/10, recomputing gate_output and up_output from saved inputs + weights ## TFlops Formula Fixes - GC-corrected multipliers: ×8/16 for recomputed components, ×6 for not - head_dim from config (not hidden_size // num_attention_heads) - Consistent seqlen via position_ids.shape[-1] across SP modes - Muon bf16 momentum: wire optimizer_dtype → muon_momentum_dtype ## Ring Attention Bug Fix - cu_seqlens computed from _original_position_ids (pre-zigzag) to avoid false document boundaries from zigzag reordering → fixes NaN loss ## Data Pipeline - Packing hash includes doc_align for ring attention cache separation - Dummy data: sequential token IDs with variable-length samples ## Benchmark - examples/benchmark/selective_gc_bench.sh: proper warmup + structured logging for accurate step time measurement * simplify: reduce duplication and minor cleanups - Extract _native_expert_forward_impl helper shared by native_expert_forward and native_expert_forward_moe_act, eliminating ~40 lines of duplicate token-sort/pad/scatter boilerplate - Add moe_recomputed property to TrainingArguments; use it in trainer and direct_train instead of inline routing-replay logic - Merge double self.modules() loop into one in base.py - Add _moe_act attribute to MoEExperts.__init__; drop getattr fallbacks - Delegate _estimate_llama_flops to _estimate_qwen2_flops (identical logic) - Remove unused culen2len/pos2culen imports from helper.py * tests: add moe_act correctness and TFLOPS benchmark tests - Forward/backward correctness for all backends (triton, quack, native) across multiple configs - Memory savings assertion - gradient_checkpointing_enable integration test - torch.compile + moe_act correctness - TFLOPS benchmark: standard vs moe_act per backend * Remove selective_gc_bench.sh example script
…fixes (#2) * Port coderforge-sft changes from tomni2: new configs, wandb improvements, dcp-only ckpt, perf fixes - Add examples/local/coderforge/ with AdamW/Muon configs for 8B/30B/32B - Remove together_coder configs/README (replaced by coderforge) - Remove enable_fsdp_offload from all dummy configs - Remove omnistore dependency; standardize on dcp checkpoint manager - Remove deprecated enable_rank0_init field - Add wandb_tags and wandb_log_interval arguments - Upload resolved config to wandb at run start - Organize wandb metrics into efficiency/training/memory sections - Add timing instrumentation to trainer setup phases - Log separate Muon/AdamW LRs to wandb - Add retry logic for weight loading in module_utils - Optimize hub dataset loading for specific splits via parquet builder - Add num_proc support for parallel dataset loading * Remove coderforge lr search scripts
…, e2e tests (#3) * Fix pipeline parallelism: padding, NaN grad skipping, loss correctness - trainer.py: pad micro-batches to fixed seq len for PP P2P buffer reuse; skip optimizer step on NaN gradients; fix PP loss reduce from SUM to MEAN - training_utils.py: refactor build_pp_loss_fn to share compiled kernel; assert uniform seq lengths in forward_backward_pp - pipeline_parallel.py: skip metadata queue pop during no_grad shape inference; disable strict shape validation for variable-length inputs - model_runner.py: fix fsdp_size=1 in PP loss fn (PP disables FSDP averaging) - muon.py: support momentum=0 (skip buffer, apply NS directly to raw grad) - Add qwen3_8b muon no-momentum and PP2 configs * Add PP e2e tests for Qwen3-8B and Qwen3-30B-A3B - qwen3_8b/test_pp.py: PP=2/FSDP=1 (2 GPU), PP=2/FSDP=4 (8 GPU) with both AdamW and Muon; validates micro-batch padding + loss normalization - qwen3_30b/test_pp.py: PP=2/EP=4/CP=4 (32 GPU) with Muon; matches production coderforge qwen3_30b_a3b_pp2_ep4_cp4_muon.yaml layout * Fix server PP: padding + reported_loss; share padding util; add server tests - training_utils.py: extract pad_micro_batches_for_pp as shared function - trainer.py: delegate _pad_micro_batches_for_pp to shared util - model_runner.py: add PP micro-batch padding; fix reported_loss (/fsdp_size was wrong after fsdp_size=1 fix — raw_total_loss is now CE_sum, not CE_sum*fsdp_size); import pad_micro_batches_for_pp - server_utils.py: add extra_config param to generate_server_config - test_pp.py (8b): add server PP tests (2 GPU + 8 GPU) - test_pp.py (30b): reduce to 8 GPUs (EP+CP folded); add server PP test * Fix PP e2e tests: dummy dataset vocab/seq capping, server weight fixture, client API - prepare_datasets.py: cap dummy dataset sample length at seq_len and use tokenizer vocab_size (was hardcoded to 151936) to avoid CUDA device-side assert on tiny models with smaller vocab - e2e_utils.py: show stdout in assert_success() for better diagnostics; decode bytes from TimeoutExpired.stdout/stderr - test_pp.py (8b/30b): use _with_weights fixtures for server tests (server requires actual weight files on disk); use _create_full_weight_client instead of raw TrainingClient() (which requires holder/model_id/base_model); increase timeouts to 600s for 8-GPU tests to avoid NCCL init contention; increase steps/lr for muon/30b tests to show clear convergence; relax min_drop_ratio from 0.05 to 0.001 for tiny models * Fix PP e2e tests: increase timeouts, fix ring-attn seq_len alignment - Increase timeout from 300s to 600s for all trainer tests to handle NCCL initialization latency when running sequentially after other tests - Fix 30B server test seq_len: use 33 instead of 32 so document length after 1-token causal shift is 32, which is divisible by 2*ringattn_size=8 * Simplify PP and FSDP gradient normalization - Always disable FSDP gradient averaging via set_gradient_divide_factor(1.0) for all non-EP FSDP modules (not just PP), removing the need for fsdp_size compensation in loss functions. EP modules are marked with _is_ep_fsdp=True to preserve their ep_size divide factor. - Remove fsdp_size param from gradient_accumulate_loss and replace build_pp_loss_fn factory with a plain module-level pp_loss_fn (compiled, always returns raw CE_sum). - Unify PP gradient normalization: trainer now explicitly scales gradients by 1/global_valid_tokens after the PP backward (matching the server's deferred optim_step normalization), eliminating the mutable _pp_global_valid_tokens closure state. - Remove global_valid_tokens param from forward_backward_pp (was unused). - Restrict supported PP schedules to GPipe and 1F1B; remove multi-stage (looped/V-style) logic from pipeline_module_split and build_pipeline_schedule. One stage per rank is now enforced with a clear error. - Cache PP schedule in server by n_microbatches to avoid rebuilding on every forward_backward call. * Enable pp_variable_seq_lengths by default; add PP benchmarks - Default pp_variable_seq_lengths=True in both Arguments and ServerArguments - Add negotiate_pp_seq_len() in training_utils: all-reduce MAX seq_len across PP group - Trainer: lazy per-seq_len schedule cache (_init_pp_schedule_cache / _get_pp_schedule) - ModelRunner: same lazy cache keyed by (n_microbatches, seq_len) - Export build_pp_stage from pipeline_parallel for cheap stage construction per seq_len - Add benchmark scripts for 8B (PP=2, 2 GPUs) and 30B-style (PP=2, EP=4, CP=4, 8 GPUs) with analysis of variable vs static padding speedup at ~65.6% fill rate * Fix PP loss logging: use op=sum over fsdp_group in _reduce_metrics The MAX all-reduce in forward_backward_pp syncs the loss only across pp_group (PP stages), not across fsdp_group (DP replicas). Different DP replicas hold different CE_sums, so using op="mean" deflated the logged loss by 1/fsdp_size when PP+DP is active. Use op="sum" to get CE_sum_total / gvt_total, matching the non-PP path where GradientAccumulateLoss already all-reduces CE_sum across DP ranks before returning the normalized scalar. Gradient normalization was already correct (manual grad.mul_(1/gvt) after FSDP's SUM all-reduce).
* Port RL training infrastructure from tomni PR #271 SGLang numerical alignment flags: - router_fp32, lm_head_fp32, rmsnorm_native, activation_native, rope_native, attention_cast_bf16 threaded from ServerArguments / model_builder through config to model layers and loss functions R3 routing weight replay: - RoutingReplay extended with record_weights/pop_forward_weights/ pop_backward_weights and has_weights property - routing_replay_handler: decode_routed_expert_logits_item, weight pre-population in fill_routing_replay - R3 SP alignment fix: pad to lcm(128, cp_size) before SP chunking (was padding to cp_size, causing routing offsets across ranks) RL loss & metrics: - IcePop hard masking (GLM-5 style): zeros gradients for IS ratio outside [1/beta, beta] via surrogate loss; icepop_beta param threaded through loss_fn_params - compute_ppo_loss returns (pg_losses, is_clipped, ratio) instead of (pg_losses, clipfrac) - Per-sample K3 KL divergence via scatter_add aggregation; LossFnOutput.k3 field API fields: - DatumInput.routed_expert_logits, LossFnOutput.k3, KillSessionRequest.reset_weights propagated through protocol layer Distributed robustness: - NCCL watchdog: three-phase error sync in runner_dispatcher (_sync_error_state) restricted to compute ops - Micro-batch count validation via all-gather before compute loop - /sync_inference_weights shorthand endpoint alias - Endpoint deduplication by (host, port) in inference_endpoints Hardening: - module_utils: sort-by-name in flush() for deterministic NCCL order, cuda.synchronize() after distribute_tensor, dispatch assertions - nccl_broadcast: handle TORCHELASTIC_USE_AGENT_STORE env var, pass device_id for proper NCCL comm init, 600s HTTP timeouts * Simplify routing_replay_handler and policy_loss - Merge decode_routed_experts_item / decode_routed_expert_logits_item into a single _decode_routing_array(dtype, log_prefix) helper; both public methods now delegate to it - Precompute lcm(128, cp_size) once per _build_per_mb_routing call instead of recomputing it inside the per-micro-batch loop - Avoid double ratio.detach() in IcePop mask computation * Add 235B 8-node config and RL launch scripts - examples/server/configs/full/qwen3_235b_a22b_8node_ep64.yaml: EP=64, Ulysses=64, FA3, Triton MoE, activation offload, all SGLang alignment flags enabled - scripts/rl_launch.sh: multi-node RL launch orchestration - scripts/launch_sgl.sh: SGLang server launcher - scripts/run_multinode_server.sh: runner_dispatcher launcher * Replace hardcoded research-common hostname prefix with NODE_PREFIX env var rl_launch.sh and launch_sgl.sh expand node IDs to hostnames by prepending NODE_PREFIX (default: ""). Set NODE_PREFIX=gpu- to get gpu-01, gpu-02, etc. from numeric IDs. * Port tomni PR #278: fix 37s/step perf regression - Replace scatter_object_list with broadcast-and-select in runner_dispatcher: all ranks locally compute their batch slice via _select_and_prepare_batches instead of waiting for rank 0 to scatter. Removes ~5s/call Gloo pickle overhead on 64 GPUs. - Remove all_gather_object micro-batch validation barrier: with broadcast-and-select all ranks deterministically compute the same micro-batch count, making the collective validation redundant (~6s/call). - Move torch._dynamo.config setup in native MoE backend from module-level to lazy init (_ensure_dynamo_configured) to avoid polluting dynamo config when native backend is imported but not used. - Fix DeepEP num_tokens_per_rdma_rank passthrough in forward and dispatch_no_grad. - Propagate execution_time through remote._execute result dict. - Pass through expert_load_summary in training_ops and request_processor. * Simplify runner_dispatcher: remove dead code, extract batch range helper - Remove _distribute_batches_to_ranks and _assign_r3_datum_offsets: both were replaced by _select_and_prepare_batches and are no longer called anywhere. - Extract _dp_batch_range(dp_rank, base_count, remainder) static helper: the balanced-distribution index formula was duplicated verbatim in _select_and_prepare_batches (for own slice) and in the datum-offset loop (for prior ranks). Both now call the helper. --------- Co-authored-by: Ashwinee Panda <apanda@together.ai>
* Add docs: installation, quickstart, local/server training, parallelism, LoRA, QLoRA, MoE, DeepEP, weight sync, dataset loading, config reference
* Reorganize docs into subfolders; add detailed per-parallelism docs
Structure:
docs/getting-started/ installation, quickstart
docs/training/ local_training, server_training, dataset_loading
docs/parallelism/ overview + dedicated docs for DP, TP, PP, EP, SP
docs/adapters/ lora, qlora
docs/moe/ overview, deepep
docs/ weight_sync, config_reference, index
Each parallelism doc covers internals in depth: mesh construction,
communication patterns, gradient handling, constraints, interaction
with other dimensions, and config examples.
* Add benchmark docs and reorganize weight_sync docs
- Add scripts/benchmark_linear_ce.py: benchmarks baseline, torch.compile,
quack (chunked_linear_cross_entropy), and liger for linear+CE speed and
memory across 4k/8k/16k sequence lengths
- Add docs/benchmarks/benchmark_linear_ce.md with results on H100
- Move docs/weight_sync.md → docs/weight_sync/overview.md
- Add docs/weight_sync/nccl_broadcast.md: deep-dive on the nccl_broadcast
backend (rank topology, transfer flow, config, limitations, extension guide)
* benchmark: add quack-gemm variant and fix liger to use LigerFusedLinearCrossEntropyLoss
- Add quack-gemm: same chunked approach as quack but replaces the two
torch.mm calls (fwd logits, dx) with quack gemm_out (CUTLASS GEMM)
- Fix liger to use LigerFusedLinearCrossEntropyLoss (transformers-level
nn.Module API) instead of the raw ops-level Function
- Update docs with full 5-column results and analysis:
quack-gemm turns out slower than quack because cuBLAS outperforms
quack CUTLASS for the tall-thin LM-head GEMM shape on H100;
quack's edge comes from the fused CE kernel, not the linear kernel
* benchmark: replace quack-gemm with LinearCrossEntropy(chunk_size=N)
Use the public nn.Module API (LinearCrossEntropy) instead of calling
chunked_linear_cross_entropy directly; drop the quack-gemm variant
* benchmark: add quack-ce vs quack-linear-ce comparison
Split quack into two methods to benchmark them independently:
- quack-ce: F.linear (full bf16 logits) + quack cross_entropy_fwd kernel
- quack-linear-ce: LinearCrossEntropy(chunk_size=4096) — chunked path
Key findings:
- quack-ce is fastest on fwd-only (no dx overhead, quack CE beats compiled)
- quack-linear-ce is fastest on fwd+bwd and matches torch.compile memory at 16k
- quack-ce uses 13.4 GB at 16k fwd+bwd (full logits retained) vs 8.6 GB for
quack-linear-ce and torch.compile (chunked, no full logit tensor)
* docs: clarify liger uses LigerFusedLinearCrossEntropyLoss from transformers API
* Move linear+CE benchmark script and doc into experiments/
scripts/benchmark_linear_ce.py → experiments/benchmark_linear_ce.py
docs/benchmarks/benchmark_linear_ce.md → experiments/benchmark_linear_ce.md
* Move linear+CE benchmark into experiments/cross_entropy/
* Add MkDocs config and GitHub Actions workflow for GitHub Pages
* docs: migrate from MkDocs to Astro Starlight
- Move all docs into docs/ subdirectory as a self-contained Astro project
- Upgrade to astro@6 and @astrojs/starlight@0.38
- Add src/content.config.ts for Astro v5+ content collections
- Add Starlight frontmatter to all markdown pages
- Update GitHub Actions workflow to build from docs/ with Node 22
* docs: light mode only, fix font visibility
* docs: change accent color from purple to blue, deploy from qingyang/docs branch
* docs: add deploy_docs.sh script and use it in CI
* docs: add contributing page and move deploy script to docs/
* docs: add testing page covering existing tests, how to run, and how to add new tests
* docs: expand config reference to cover all local and server arguments with accurate defaults
* docs: add manual deployment commands to contributing page
* docs: split testing into subsections (overview, existing tests, running, adding)
* docs: collapse all sidebar sections except Getting Started by default
* Add NCCL speed test benchmark
* fix: set site URL for GitHub Pages; add run_e2e.sh script and update docs
* docs: remove run_distributed_tests.sh, use torchrun directly in docs
* docs: fix distributed tests — work with plain pytest, torchrun spawned internally
* docs: add development guide
* docs: rename Testing to Tests in sidebar
* docs: group Development Guide and Contributing to Docs under Contributing section
* docs: redesign home page with feature cards and navigation links
* docs: style hero title with gradient, tighten intro paragraph spacing
* docs: reduce whitespace on home page
* docs: move Explore the docs section above Features
* docs: split config reference into separate local and server pages
* docs: add xorl-client install, client usage, and Tinker API compatibility to server training
* docs: split Training into separate Local Training and Server Training sections
* docs: move Weight Sync under Server Training section
* docs: nest Weight Sync as sub-group under Server Training
* docs: add SVG diagrams and source code references to all doc sections
* fix: remove HTML comments from MDX SVGs (invalid JSX)
* fix: use absolute paths for cross-page links to avoid 404s
* docs: fix SVG artifacts, add distribution figures, expand MoE and checkpoint docs
- Task 1: Fix overview.mdx SVG — split long world_size formula text into two lines at y=302/314, expand viewBox height to 326
- Task 2: Rename lora.mdx SVG marker IDs from a2/ag to lora-arrow/lora-arrow-gray (avoids generic ID collisions)
- Task 3: Add data layout distribution SVGs to all 5 parallelism files (dp-dist, tp-dist, pp-dist, sp-dist, ep-dist marker IDs; no HTML comments)
- Task 4: Expand moe/overview.mdx with GKN layout section, updated kernels table (EP/compile/LoRA columns), LoRA-on-experts subsection, torch.compile guidance, and moe_checkpoint_method comparison table
- Task 5: Create training/loss_functions.mdx covering causallm_loss, policy_loss, importance_sampling_loss, per_token_ce, compiled CE backend, vocab-parallel CE for TP, and loss flow SVG
- Task 6: Create training/checkpointing.mdx covering DCP format, save/load config, broadcast vs all_ranks weight loading SVG, FSDP sharding details
- Task 7: Add Loss Functions and Checkpointing entries to astro.config.mjs Local Training section
* docs: move Loss Functions to top level (applies to both local and server training)
* docs: significantly expand MoE docs — R3 routing replay, GKN layout, backends, hybrid LoRA, DeepEP tuning
* docs: split MoE into focused subsections — router, kernels, EP, LoRA
* docs: add RL Training section — overview, IcePop, TIS, R3, xorl-sglang; move Server Training and Loss Functions under RL
* docs: flatten Server Training and RL Training into single section
* docs: remove MaxRL from RL Orchestrator box in architecture diagram
* docs: reorganize files — server-training/ dir, loss-functions at top level, fix all internal links
* docs: link EP parallelism page to MoE, remove duplicates from moe/expert-parallelism
* docs: add Server Architecture and Trainer Architecture pages with system design, APIs, and data pipeline details
* docs: add zigzag document padding requirement, packing algorithms (sequential/FFD) to both training pages
* docs: expand Quack backend — CuTe DSL, SM90/SM100 configs, pingpong, autotuning, feature matrix, DeepEP pairing
* docs: fix weight loading SVG — proper spacing, readable font sizes, distinct rank boxes
* docs: add checkpoint optimization details — multi-stream DMA, async save, prefetch, batched broadcast, EP filtering
* docs: remove all MaxRL references
* docs: fix weight loading SVG — proper box spacing, no overlaps
* docs: move Full API Reference to its own page under Server Training
* docs: add Gradient Accumulation sub-page under Loss Functions — token normalization, DP/CP/grad-accum groups, FSDP divide factor
* docs: add server training deferred normalization to gradient accumulation page
* docs: add distillation guide, supported models page, pp_variable_seq_lengths, fix missing config params
* docs: remove all distillation references
* docs: rename Sequence Parallelism to Context Parallelism throughout
* docs: note that SP and CP are interchangeable in context parallelism page
* ci: deploy docs from main branch
* docs: fix inaccuracies across documentation vs code
- API reference: rename save_state/load_state to save_weights/load_weights,
remove non-existent save_lora_only/load_lora_adapter endpoints, fix HTTP
methods (list_checkpoints and weights_info are POST not GET), fix inference
endpoint paths (no /api/v1/ prefix), add missing training_runs endpoint
- Server config: remove duplicate reshard_after_forward entry
- Local config: remove non-existent correction_rank and correction_ns_steps
fields from lora section
- Local training: remove invalid max_steps=-1, fix wandb_run_name to
wandb_name, fix log_interval to wandb_log_interval, fix
activation_gpu_limit unit (GB not percentage), fix checkpoint path format
- Models: MoE weight conversion is automatic at load time, not a separate
preprocessing step (xorl.tools.moe_merge does not exist)
- Loss functions: fix source reference to compiled_cross_entropy.py and
vocab_parallel_cross_entropy.py (cross_entropy.py does not exist)
- QLoRA: remove Quantization Error Correction section (correction_rank and
correction_ns_steps fields do not exist in LoRAArguments)
- Propagate endpoint and MoE conversion fixes to server-training/overview,
adapters/lora, and moe/overview pages
* docs: fix world_size formula, LoRA shapes, dataset types, quickstart API format
- parallelism/overview: remove EP from world_size formula — EP is folded
onto the same GPU axis as CP, not a separate multiplier. The code
(arguments.py, parallel_state.py) computes world_size = dp × tp × pp ×
ringattn × ulysses with no ep factor.
- moe/lora: fix LoRA adapter shapes — lora_A is [E, in_features, r] not
[E, r, in_features], lora_B is [E, r, out_features] not [E, out_features, r].
Verified against src/xorl/models/layers/moe/lora.py parameter creation.
- moe/lora: fix source paths — LoRA base module is at src/xorl/lora/
not src/xorl/models/layers/lora/, QLoRA is at src/xorl/qlora/ not
src/xorl/models/layers/qlora/
- dataset_loading: remove plaintext and conversation dataset types — code
only supports type='tokenized' (raises NotImplementedError otherwise)
- installation: fix PyTorch version 2.9+ → 2.10+ (pyproject.toml pins 2.10.0)
- quickstart: fix forward_backward API request format — uses
forward_backward_input.data[].model_input/loss_fn_inputs, not batches[]
- adapters/qlora: fix lora_alpha default 32 → 16 (matches LoRAArguments)
* docs: fix EP/world_size errors in parallelism docs
- pipeline_parallelism Example 2: fix GPU count 32 → 8 and remove EP
from world_size formula. The referenced config
(qwen3_30b_a3b_pp2_ep4_cp4_muon.yaml) uses PP=2, ringattn=4, dp=1
which gives world_size=8, not 32. EP=4 is folded onto each PP stage's
4 ranks.
- pipeline_parallelism: fix pad_to_multiple_of default 1 → 128 (code
default in arguments.py)
- expert_parallelism: fix two EP+PP examples that incorrectly multiplied
EP into GPU count. Remove "Wait —" draft artifact that leaked into
published doc.
* docs: fix installation verify commands and uv sync flags
- Fix flash attention verify: import flash_attn → import flash_attn_interface
(FA3 package installs as flash_attn_interface, not flash_attn). Add separate
FA4 verify via flash_attn.cute.
- Remove --no-build-isolation from uv sync (not needed, matches CLAUDE.md)
* docs: fix quickstart server example and config path
- Fix server config path: qwen3_8b.yaml → qwen3_8b_full.yaml (actual filename)
- Fix server training example: all training endpoints use a two-phase async
pattern (POST returns request_id, poll retrieve_future for result). Previous
example suggested forward_backward returns the loss directly.
- Fix lora_alpha: 32 → 16 in LoRA and QLoRA examples (matches code default
and actual example configs)
* docs: fix log_format description in local training
structured format produces key=value lines, not JSON
* docs: fix server training overview, loss function names, endpoint paths
- overview: fix server config path qwen3_8b.yaml → qwen3_8b_full.yaml
- overview: fix --api-port default 5555 → auto (None), --operation-timeout
3600 → 1800, --master-addr localhost → 127.0.0.1
- overview: fix optim_step AdamParams field "epsilon" → "eps"
- overview: fix loss_fn name importance_sampling_loss → importance_sampling
(matches LOSS_REGISTRY key)
- overview: remove per_token_ce from loss function table (not a registered
loss_fn — per-token outputs are a return mode, not a separate function)
- overview: fix inference endpoint paths /api/v1/add_inference_endpoint →
/add_inference_endpoint (no prefix), same for remove
- overview: fix sleep/wake paths /api/v1/sleep → /sleep
- overview: fix Tinker compat table: weights_info is POST not GET,
list_training_runs → training_runs
- loss-functions: rename importance_sampling_loss → importance_sampling
throughout, update per_token_ce section to describe per-token outputs
as a return mode rather than a standalone loss function
- sglang: fix /api/v1/add_inference_endpoint → /add_inference_endpoint
* docs: fix server config CLI syntax
- Fix launch command: positional config.yaml → --mode auto --config config.yaml
(launcher requires --config flag and --mode parameter)
- Fix CLI override syntax: bare --key → --server.key prefix
(launcher uses parse_server_overrides which requires --server. prefix)
* fix: correct metadata, type annotations, and typos in argument definitions
arguments.py:
- sample_packing_method: remove 'none' from help text (validation only
accepts 'sequential' or 'multipack')
- ds_type: add missing 'text' to help text (validation accepts it)
- encoders: widen Literal type from ["image"] to ["image", "video", "audio"]
to match __post_init__ validation
- Fix typo: suppoerted_encoder_types → supported_encoder_types
- Fix typo: defult → default (lr help text)
- Fix typo: CUDA_LUANCH_BLOCKING → CUDA_LAUNCH_BLOCKING (assertion message)
- Fix typo: CUDA_LAUNCH_BLOCK → CUDA_LAUNCH_BLOCKING (help text)
- Fix typo: Direction → Directory (profile_trace_dir help text)
server_arguments.py:
- encoders: widen Literal type to match arguments.py
- quant_format: add missing 'nf4' to help text (QLoRA code supports it)
* docs: usability improvements and correctness fixes across 13 pages
Usability improvements:
- Add "Choosing a Training Mode" decision table to quickstart
- Add tokenized dataset format example to dataset_loading
- Expand meta device explanation in local_training
- Add "When to Use Each Dimension" guidance to parallelism overview
- Add loss function decision table to rl-training
- Add TL;DR note to gradient-accumulation
- Add field interaction constraints to config reference
- Add MoE LoRA cross-links to lora and qlora pages
- Add async pattern note to API reference
- Add "Advanced" separator to checkpointing
- Add "Next Steps" link to installation
- Fix broken relative link in nccl-broadcast
Correctness fixes found during audit:
- Fix ring attention constraint: 2×ringattn×ulysses (not just 2×ringattn)
- Fix cache path: {dataset_prepared_path}/{hash}/ (not {output_dir}/...)
- Fix quant_group_size: single default of 16, not format-dependent
- Remove reference to nonexistent uv.lock file
- Fix EP/CP relationship: EP uses separate mesh, not folded with CP
---------
Co-authored-by: Ashwinee Panda <apanda@together.ai>
Replace the circles+text logo with the new standalone gradient wordmark and set replacesTitle: true since the logo now includes the title.
* docs: add DeepEP multi-node prerequisites and cluster setup - Document nvidia_peermem requirement for GPUDirect RDMA — without it NVSHMEM cannot register GPU buffers with IB HCAs and DeepEP crashes with SIGABRT at the first dispatch - Document IBGDA driver settings (NVreg_EnableStreamMemOPs, PeerMappingOverride) with initramfs rebuild and reboot steps - Add troubleshooting entries for SIGABRT/num_recv_tokens=-1 and IBGDA init failures - Add 2-node EP16 dummy benchmark configs (deepep and alltoall) - Add slurm_train_30b_2node.sh launch script with corrected NVSHMEM lib path detection (use __path__ instead of __file__ for namespace packages) - Add enable_ibgda_reboot.sh for per-node IBGDA kernel module setup * docs: remove DeepEP install section from README * docs: consolidate DeepEP install and multi-node prerequisites under single section * docs: point DeepEP install to upstream repo instead of internal wheel * docs: remove multi-node setup section from installation page * docs: add link to installation guide in quickstart * docs: simplify README, link to docs for features and details * revert: remove examples and scripts from PR * docs: use standard GitHub Pages URL (org/repo format) * docs: update README — XoRL logo, remove tagline * docs: add overview section and improve README layout * docs: add emojis to README section headings * docs: fix clone URL to togethercomputer/xorl-internal * docs: rename DEVELOPMENT.md to CONTRIBUTING.md, add contributing section to README * docs: add supported models section to README * docs: add emojis to header nav links * tests: add DeepEP vs AllToAll correctness test (forward + backward, single- and cross-node)
Add Qwen3.5 and Qwen3.5-MoE model support with native FLA context parallelism Model support: - Qwen3_5ForCausalLM (dense) and Qwen3_5MoeForCausalLM (MoE) with hybrid full_attention / linear_attention (GatedDeltaNet) layer architecture - Checkpoint handlers for HF weight loading: QKV merge, gate/up merge, fused-to-split linear attention weight mapping, EP slicing for MoE - Config adapters (from_hf_config) for both dense and MoE variants - TP/EP/PP parallelization plans Linear attention stack (adapted from flash-linear-attention, MIT license): - GatedDeltaNet layer with chunk and fused_recurrent modes - Triton kernels for chunk_delta_h, chunk_o, wy_fast, cumsum, solve_tril - ShortConvolution with causal depthwise conv and CP-aware forward/backward - Native FLA context parallelism for Ulysses sequence-parallel training, with all-gather based inter-rank state propagation in forward and backward MFU calculation: - Dedicated flops estimators for xorl_qwen3_5 and xorl_qwen3_5_moe that correctly count O(n^2) attention only for full_attention layers, account for attention gate (doubled q_proj), shared expert, and GatedDeltaNet projection FLOPs for linear_attention layers Tests: - Registry, config from_hf_config, GatedDeltaNet backward shape regression, Ulysses CP positive/negative smoke, CP vs single-GPU equivalence Configs: - CoderForge training configs for Qwen3.5-35B-A3B (AdamW and Muon) - Dummy smoke test configs for 35B-A3B and 4B
qywu
added a commit
that referenced
this pull request
Mar 23, 2026
…fixes (#2) * Port coderforge-sft changes from tomni2: new configs, wandb improvements, dcp-only ckpt, perf fixes - Add examples/local/coderforge/ with AdamW/Muon configs for 8B/30B/32B - Remove together_coder configs/README (replaced by coderforge) - Remove enable_fsdp_offload from all dummy configs - Remove omnistore dependency; standardize on dcp checkpoint manager - Remove deprecated enable_rank0_init field - Add wandb_tags and wandb_log_interval arguments - Upload resolved config to wandb at run start - Organize wandb metrics into efficiency/training/memory sections - Add timing instrumentation to trainer setup phases - Log separate Muon/AdamW LRs to wandb - Add retry logic for weight loading in module_utils - Optimize hub dataset loading for specific splits via parquet builder - Add num_proc support for parallel dataset loading * Remove coderforge lr search scripts
qywu
added a commit
that referenced
this pull request
Apr 18, 2026
…fixes (#2) * Port coderforge-sft changes from tomni2: new configs, wandb improvements, dcp-only ckpt, perf fixes - Add examples/local/coderforge/ with AdamW/Muon configs for 8B/30B/32B - Remove together_coder configs/README (replaced by coderforge) - Remove enable_fsdp_offload from all dummy configs - Remove omnistore dependency; standardize on dcp checkpoint manager - Remove deprecated enable_rank0_init field - Add wandb_tags and wandb_log_interval arguments - Upload resolved config to wandb at run start - Organize wandb metrics into efficiency/training/memory sections - Add timing instrumentation to trainer setup phases - Log separate Muon/AdamW LRs to wandb - Add retry logic for weight loading in module_utils - Optimize hub dataset loading for specific splits via parquet builder - Add num_proc support for parallel dataset loading * Remove coderforge lr search scripts
This was referenced Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan