[RFC] NextTransformer scaling plans: 200B/400B tokens, 4K context, and Dense/MoE configurations

[Selen-Suyue]

We are planning the next scaling stage of NextTransformer and would welcome community feedback before finalizing the training recipe.

Our current benchmark uses 49.152B training tokens and a 2,048-token context window. For the next stage, we are considering 200B-token and 400B-token pretraining runs, with both Dense and MoE models, at a 4,096-token sequence length.

We would especially appreciate suggestions on data composition, model configurations, and evaluation beyond CORE. The configurations below are starting points for discussion.

1. Proposed training setup

• Primary dataset: ClimbMix.
• Training budgets: 200B and 400B consumed training tokens, measured with our tokenizer.
• Sequence length: 4,096 tokens for the main pretraining stage.
• Tokenizer: Retain the existing 32,768-token tokenizer.
• Model families: Dense and sparse MoE.
• Evaluation: Retain validation BPB and CORE-22, while considering additional evaluations that can reveal benefits from the longer context.

ClimbMix would remain the foundation of the training mixture. Whether to supplement it with other sources is one of the main questions in this proposal. Longer-context continuation beyond 4K is a possible subsequent stage.

2. Candidate model configurations

Revised on Nov 5, 2026: Dense @ 400B now uses 48 layers, a 3,072 hidden dimension, and 24 attention heads (5.44B block parameters), replacing 50 layers, 3,200, and 25 heads (6.14B). This update explains the change and the MoE details we are considering.

Configuration	Dense @ 200B	MoE @ 200B	Dense @ 400B	MoE @ 400B
Transformer layers	40	32	48	40
Hidden dimension	2,560	2,048	3,072	2,560
Attention heads	20	16	24	20
Head dimension	128	128	128	128
Dense FFN hidden dimension	10,240	8,192, first layer only	12,288	10,240, first layer only
Expert FFN hidden dimension	—	512	—	640
Routed experts per MoE layer	—	256	—	256
Routed experts selected per token	—	8	—	8
Shared experts per MoE layer	—	1	—	1
Approx. total block parameters¹	3.15B	17.3B	5.44B	34.0B
Approx. active block parameters per token¹	3.15B	1.17B	5.44B	2.28B

The starting architecture follows our nanochat-derived backbone: full causal multi-head attention, ReLU² FFNs, and Muon + AdamW optimization. In the MoE models, the first layer uses a dense FFN; subsequent layers use MoE FFNs. The shared expert has the same hidden dimension as each routed expert.

¹ These estimates cover the Transformer blocks, including the MoE routers. They exclude token embeddings, value-embedding tables, and the output head, and therefore are not full-model parameter counts. Retaining the existing value embeddings would add substantial parameters; we welcome feedback on that choice as well.

The Dense and MoE candidates at each token budget are not compute-matched. The proposed MoE models have smaller active parameter counts and larger total capacity. We want to investigate their quality–compute trade-off, and welcome suggestions on whether an additional matched-compute comparison should be included. Final compute accounting would include attention, routing, the output head, and the rest of the training workload.

3. Questions for the community

A. Is ClimbMix sufficient for 4K pretraining, or should we mix in other data?

Our current understanding is that ClimbMix contains many short documents. Packing them into 4K sequences improves utilization, but does not by itself provide continuous 4K document context.

• Would you keep the main stage entirely on ClimbMix, upsample its longer documents, or introduce additional datasets?
• Which sources would you recommend for useful longer-range dependencies—for example, long-form web text, books, scientific articles, or code?
• What mixing ratio, measured by training tokens, would you suggest? Should those sources be introduced from the beginning or during a later stage?

Dataset links, length statistics, and evidence from comparable training budgets would be particularly helpful. We would also appreciate practical advice on preserving document continuity during packing.

B. Are the proposed Dense and MoE configurations appropriate for these budgets?

• Are the depth–width choices and parameter scales sensible for 200B and 400B tokens?
• Is 256 routed experts with top-8 routing and one shared expert too sparse for these training budgets? Would 64 or 128 experts be a stronger starting point?
• Would you change the expert size, shared-expert design, routing/load-balancing strategy, or backbone components?
• Should we prioritize equal-token comparisons, equal-compute comparisons, or include both?

Concrete alternative configurations and results at similar scales would be especially valuable. We are interested in suggestions that can be tested with smaller pilot runs before committing to the full budgets.

C. What should we evaluate beyond CORE?

We plan to retain CORE-22 for general capability tracking. For additional context-sensitive evaluation, our initial shortlist is:

Candidate	Proposed use	Why we are considering it
HELMET — RAG tasks	Start with Natural Questions and TriviaQA; potentially add HotpotQA	English evidence-based QA, with few-shot support for evaluating base models. Tests whether models can use relevant information among distracting passages.
RULER	Start with single-target, multi-target, and multi-value retrieval	Configurable lengths and retrieval difficulty help diagnose whether information remains accessible across the context window.

We would begin with controlled 2K and 4K evaluations, and add longer lengths if we train extended-context checkpoints. Any adaptations to the official benchmark settings would be documented.

Because these are pretraining checkpoints, we would first verify that models can perform the tasks at short lengths, so that instruction-following or formatting failures are not mistaken for context-length failures. We would also separate gains from longer-context training from gains obtained simply by allowing more input at evaluation time.

Are these two additions appropriate for ClimbMix-trained base models at our proposed scales? Are there other benchmarks that would provide a more reliable or informative signal?

How to contribute

Feedback on any one of these areas is welcome. A useful recommendation could include:

• The change you would make.
• Why it should help at the proposed scale.
• A supporting paper, dataset, implementation, or experimental result.
• A small ablation that could validate the idea.

We would like to use this discussion to identify the most informative pilot experiments and refine the scaling recipe before launching the full runs.

[Selen-Suyue]

Update (Nov 5): revised Dense @ 400B configuration

Thanks to everyone who has shared feedback so far. After reviewing recent open scaling work, including Marin's Delphi scaling suite, Marin's MoE pretraining-efficiency report, and Microsoft AI's MAI-Thinking-1 report, we are revising one of the four candidate configurations. The other three are unchanged.

Configuration	Dense @ 400B (original)	Dense @ 400B (revised)
Transformer layers	50	48
Hidden dimension	3,200	3,072
Attention heads	25	24
Head dimension	128	128
Dense FFN hidden dimension	12,800	12,288
Approx. block parameters¹	6.14B	5.44B
Training tokens per block parameter	~65	~74

¹ Same convention as the original table: Transformer blocks only, excluding token embeddings, value-embedding tables, and the output head.

Why we changed it

• Hardware-friendly shape. A hidden size of 3,200 is not a multiple of 512, and 25 attention heads cannot be split evenly across 2-, 4-, or 8-way tensor parallelism. A hidden size of 3,072 with 24 heads avoids both problems.
• One shape rule across all candidates. The revision keeps d_model = 64 × layers, the depth-to-width ratio used by the other three candidates. The 200B → 400B dense step now changes scale, not shape.
• Lower cost. By our estimate, training FLOPs per token drop by about 11%. This estimate includes 4K causal attention and the output head.

The trade-off we are accepting

Both dense tiers are intentionally trained beyond compute-optimal: about 64 and 74 tokens per parameter, compared with roughly 20 under Epoch AI's replication of the Chinchilla fit. Under that fit, the extra training costs about 12–16% in compute-equivalent loss. We accept this cost in exchange for smaller models that are cheaper for the community to evaluate and fine-tune, full use of the planned token budgets, and a consistent tokens-per-parameter ratio across tiers.

Still open: depth vs. width

Recent recipes use wider, shallower shapes than ours. MAI-Base-1's scaling ladder uses d_model ≈ 85 × layers, and Delphi's depth rule gives about 100 × layers at these widths. We plan to compare our 64× ratio with a ~90× variant in small-scale pilots. If the wider shape matches or beats ours in both loss and throughput, we will switch both dense tiers together. Results or suggestions from comparable scales are very welcome.

MoE: what we are considering (feedback welcome)

• Keep high sparsity. We plan to keep 256 routed experts with top-8 routing plus one shared expert. Marin (64 → 256 experts) and MAI (256 → 1,024 experts) both report gains from higher sparsity that grow with compute, and MAI finds that layouts with MoE in every layer rely on shared experts.
• Make load balancing explicit. The candidates are Quantile Balancing, or a balancing loss aggregated over the global batch with dropless routing. We would also add a router z-loss and zero-initialize the attention outputs, which MAI reports reduces early routing imbalance.
• Benchmark a coarser expert layout. The variant uses 128 experts of width 1,024, top-4 routing, and one shared expert. Total parameters per layer stay nearly the same (541M vs. 539M), active parameters rise by about 11%, and the expert matrix multiplications double in size. We will choose between the two layouts based on measured throughput on our cluster.