Clover-35B-A3B-Preview

The first diffusion language model from DiffuSpace.

Introduction · Performance · Quick Start · Inference Backends · Fine-Tuning

We introduce Clover-35B-A3B-Preview, the first model released by DiffuSpace. Clover delivers state-of-the-art performance among open-source diffusion language models at release on the benchmarks reported in our launch blog.

Built on Qwen3.6-35B-A3B, Clover combines a text-only mixture-of-experts backbone with block diffusion generation:

• 35B total parameters, approximately 3B active per token. A hybrid Gated DeltaNet (GDN) / full-attention MoE architecture.
• Bidirectional attention within each block. Additional learned GDN parameters support block diffusion while preserving a causal prefix.
• 128K context. Up to 131,072 tokens across the prompt and generated response.

See the launch blog for benchmark results, evaluation settings, and comparisons.

Install PyTorch with CUDA support, then the Hugging Face dependencies. This example uses Transformers 5.19.0 and PyTorch 2.10 or later.

pip install "transformers==5.19.0" accelerate safetensors

The example runs on one GPU with enough memory for the BF16 weights (approximately 69 GB) plus inference state.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Diffuspace/Clover-35B-A3B-Preview"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="cuda:0",
).eval()

messages = [{"role": "user", "content": "What is diffusion?"}]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True,
)
inputs = tokenizer(
    prompt, add_special_tokens=False, return_tensors="pt"
).to("cuda:0")

output = model.generate(**inputs, max_new_tokens=24576)
completion = output[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(completion, skip_special_tokens=True))

Generation defaults are included in the checkpoint. For decoding options and configuration details, see the Hugging Face model card.

Our SGLang implementation supports both Block diffusion denoising and Self-speculative decoding for Clover. The llama.cpp implementation supports Self-speculative decoding. Backend code and setup guides will be added to this repository.

Coming soon. SFT code and training instructions will be released in sft/.

The model is distributed under Apache-2.0. See the model license and attributions for details.

Clover builds on Qwen3.6-35B-A3B. We thank the Qwen, Hugging Face Transformers, SGLang, and llama.cpp communities for their open-source work.