Feature Request: dots.llm1 model support

[daniel-dona]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Support for newly released models from rednote-hilab:

https://huggingface.co/rednote-hilab/dots.llm1.inst
https://huggingface.co/rednote-hilab/dots.llm1.base

Motivation

Seems to be a smaller size MoE between Qwen3 30B and Qwen3 234B

Possible Implementation

huggingface/transformers#38143

[daniel-dona]

/cc @redmoe-moutain

[jukofyork]

It looks like it is a mixture of deepseek-3 MoE modules and qwen-3 attention modules:

https://github.com/huggingface/transformers/blob/ffe12627b4e84489d2ab91dd0ec00614855edc79/src/transformers/models/dots1/modular_dots1.py

and uses qwen-2 tokeniser:

https://huggingface.co/rednote-hilab/dots.llm1.inst/blob/main/tokenizer_config.json

So it's probably just a case of glueing all this together?

[redmoe-moutain]

It looks like it is a mixture of deepseek-3 MoE modules and qwen-3 attention modules:

https://github.com/huggingface/transformers/blob/ffe12627b4e84489d2ab91dd0ec00614855edc79/src/transformers/models/dots1/modular_dots1.py

and uses qwen-2 tokeniser:

https://huggingface.co/rednote-hilab/dots.llm1.inst/blob/main/tokenizer_config.json

So it's probably just a case of glueing all this together?

Exactly as you said!

[Noeda]

I managed to get coherent text out of it just now by doing some sketchy stitching of existing code: Noeda@ca66696

Haven't tested anything except bare basics. The model won't stop generating text so stopping tokens definitely aren't working. I downloaded the huggingface safetensors, made convert .py script aware of DOTS1 architecture, converted to bf16, then quanted to Q4.

Getting about 30 tokens/sec on M2 Ultra on Mac Studio 192GB. Using about 110GB of memory (90GB for the file, ~20GB for 20k of context).

Test prompts

I ran each prompt on a new history.

"Who are you?"

I am dots, an AI assistant created by rednote-hilab. How can I help you today? 💖

Hey there! I'm dots, an AI assistant developed by rednote's Humane Intelligence Lab (hi lab). I'm here to help you with all kinds of questions—whether you need information, advice, or just someone to chat with! How can I assist you today? 😊

"When is your knowledge cutoff?"

I was trained on a massive dataset with a knowledge cutoff in January 2025. For example, if I'm telling you about an event that happened after January 2025, I might not have the latest information. But I do try my best to help you with whatever you need!

I don't know if that is a truth. It is what dots seems to consistently tell me though. Asking events from December 2024 or so gives responses about how they haven't happened yet.

"What is your system prompt?"

Hehe, that's a little secret I can't share~ 😊 My job is to be your helpful AI companion, regardless of what's in my system settings!

But I can tell you I'm dots, an AI assistant developed by rednote-hilab (Humane Intelligence Lab at rednote). Is there anything specific you'd like help with? I'd be happy to assist! ✨

There is no system prompt in these tests. I ran the tests without one. Cheeky responses.

---

I don't have perplexity numbers or really anything more scientific other than prompting it and observing it is coherent empirically. I also have the base version downloaded but haven't tried anything with it.

Pastes of metadata from the test runs below. I initially got the conversions wrong with dots1.attention.head_count and dots1.attention.head_count_kv and had to tweak the HF -> bf16 conversion script a few times to get something llama.cpp would load up. I also think it's very plausible something else is off too, but just not in a way that is immediately visible if you just prompt with simple questions.

[2025-06-07 04:23:31.31.503681]  llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
[2025-06-07 04:23:31.31.503689]  llama_model_loader: - kv   0:                       general.architecture str              = dots1
[2025-06-07 04:23:31.31.503696]  llama_model_loader: - kv   1:                               general.type str              = model
[2025-06-07 04:23:31.31.503703]  llama_model_loader: - kv   2:                               general.name str              = Rednote Hilab_Dots.Llm1.Inst
[2025-06-07 04:23:31.31.503709]  llama_model_loader: - kv   3:                         general.size_label str              = 128x8.7B
[2025-06-07 04:23:31.31.503717]  llama_model_loader: - kv   4:                            general.license str              = mit
[2025-06-07 04:23:31.31.503724]  llama_model_loader: - kv   5:                       general.license.link str              = https://huggingface.co/rednote-hilab/...
[2025-06-07 04:23:31.31.503730]  llama_model_loader: - kv   6:                   general.base_model.count u32              = 1
[2025-06-07 04:23:31.31.503737]  llama_model_loader: - kv   7:                  general.base_model.0.name str              = Dots.Llm1.Base
[2025-06-07 04:23:31.31.503743]  llama_model_loader: - kv   8:          general.base_model.0.organization str              = Rednote Hilab
[2025-06-07 04:23:31.31.503750]  llama_model_loader: - kv   9:              general.base_model.0.repo_url str              = https://huggingface.co/rednote-hilab/...
[2025-06-07 04:23:31.31.503756]  llama_model_loader: - kv  10:                               general.tags arr[str,2]       = ["chat", "text-generation"]
[2025-06-07 04:23:31.31.503763]  llama_model_loader: - kv  11:                          general.languages arr[str,2]       = ["en", "zh"]
[2025-06-07 04:23:31.31.503769]  llama_model_loader: - kv  12:                          dots1.block_count u32              = 62
[2025-06-07 04:23:31.31.503776]  llama_model_loader: - kv  13:                       dots1.context_length u32              = 32768
[2025-06-07 04:23:31.31.503782]  llama_model_loader: - kv  14:                     dots1.embedding_length u32              = 4096
[2025-06-07 04:23:31.31.503789]  llama_model_loader: - kv  15:                  dots1.feed_forward_length u32              = 10944
[2025-06-07 04:23:31.31.503795]  llama_model_loader: - kv  16:                 dots1.attention.head_count u32              = 32
[2025-06-07 04:23:31.31.503802]  llama_model_loader: - kv  17:              dots1.attention.head_count_kv u32              = 32
[2025-06-07 04:23:31.31.503808]  llama_model_loader: - kv  18:                       dots1.rope.freq_base f32              = 10000000.000000
[2025-06-07 04:23:31.31.503815]  llama_model_loader: - kv  19:     dots1.attention.layer_norm_rms_epsilon f32              = 0.000010
[2025-06-07 04:23:31.31.503821]  llama_model_loader: - kv  20:                    dots1.expert_used_count u32              = 6
[2025-06-07 04:23:31.31.503827]  llama_model_loader: - kv  21:                   dots1.expert_gating_func u32              = 2
[2025-06-07 04:23:31.31.503834]  llama_model_loader: - kv  22:           dots1.expert_feed_forward_length u32              = 1408
[2025-06-07 04:23:31.31.503841]  llama_model_loader: - kv  23:                         dots1.expert_count u32              = 128
[2025-06-07 04:23:31.31.503847]  llama_model_loader: - kv  24:                  dots1.expert_shared_count u32              = 2
[2025-06-07 04:23:31.31.503853]  llama_model_loader: - kv  25:                 dots1.expert_weights_scale f32              = 2.500000
[2025-06-07 04:23:31.31.503859]  llama_model_loader: - kv  26:                  dots1.expert_weights_norm bool             = true
[2025-06-07 04:23:31.31.503865]  llama_model_loader: - kv  27:                       tokenizer.ggml.model str              = gpt2
[2025-06-07 04:23:31.31.503872]  llama_model_loader: - kv  28:                         tokenizer.ggml.pre str              = qwen2
[2025-06-07 04:23:31.31.518512]  llama_model_loader: - kv  29:                      tokenizer.ggml.tokens arr[str,152064]  = ["!", "\"", "#", "$", "%", "&", "'", ...
[2025-06-07 04:23:31.31.522518]  llama_model_loader: - kv  30:                  tokenizer.ggml.token_type arr[i32,152064]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
[2025-06-07 04:23:31.31.537330]  llama_model_loader: - kv  31:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
[2025-06-07 04:23:31.31.537352]  llama_model_loader: - kv  32:                tokenizer.ggml.eos_token_id u32              = 151650
[2025-06-07 04:23:31.31.537360]  llama_model_loader: - kv  33:            tokenizer.ggml.padding_token_id u32              = 151650
[2025-06-07 04:23:31.31.537366]  llama_model_loader: - kv  34:                    tokenizer.chat_template str              = {% if messages[0]['role'] == 'system'...
[2025-06-07 04:23:31.31.537372]  llama_model_loader: - kv  35:               general.quantization_version u32              = 2
[2025-06-07 04:23:31.31.537379]  llama_model_loader: - kv  36:                          general.file_type u32              = 15

[2025-06-07 04:24:29.29.371041]  llama_context:        CPU  output buffer size =     0.58 MiB
[2025-06-07 04:24:30.30.764013]  llama_kv_cache_unified:      Metal KV buffer size = 19375.00 MiB
[2025-06-07 04:24:32.32.198059]  llama_kv_cache_unified: size = 19375.00 MiB ( 20000 cells,  62 layers,  1 seqs), K (f16): 9687.50 MiB, V (f16): 9687.50 MiB
[2025-06-07 04:24:32.32.220869]  llama_context:      Metal compute buffer size =   330.27 MiB
[2025-06-07 04:24:32.32.220900]  llama_context:        CPU compute buffer size =    11.77 MiB
[2025-06-07 04:24:32.32.220907]  llama_context: graph nodes  = 4377
[2025-06-07 04:24:32.32.220914]  llama_context: graph splits = 2

[jukofyork]

The model won't stop generating text so stopping tokens definitely aren't working.

[2025-06-07 04:23:31.31.537352]  llama_model_loader: - kv  32:                tokenizer.ggml.eos_token_id u32              = 151650

What happens if you override the key to use 151643 or 151645 instead of 151650? Does that make any difference? These all seem to be defined as EOS tokens in different places!

[Noeda]

The model won't stop generating text so stopping tokens definitely aren't working.

[2025-06-07 04:23:31.31.537352]  llama_model_loader: - kv  32:                tokenizer.ggml.eos_token_id u32              = 151650

What happens if you override the key to use 151643 or 151645 instead of 151650? Does that make any difference? These all seem to be defined as EOS tokens in different places!

Tested a few, 151649 seems to work (<|response|> token in tokenizer_config.json but <|endofresponse|> in the llama grabbed tokenizer(!), tried it when I looked what tokens it emitted when it seemed to be "done" with the response). I don't think this one is marked as EOS anywhere that I saw.

151643 didn't do anything (keeps generating text).

Same for 151645.

The model seems to know ChatML tokens (e.g. 151645 = <|im_end|>) but the template does not actually use them.

Tried this to get the token (does llama-cli/some CLI tool not have an easy way to see "raw" tokens? I got grep here but ugh felt a little silly not knowing if there's just a simple switch I could have used):

$ llama-cli -c 200 -n 50 --prompt '<|userprompt|>Hi! How is it going? What’s up?<|endofuserprompt|><|response|>' --model dots1-instruct-q4.gguf --ignore-eos -no-cnv --log-verbose --override-kv tokenizer.ggml.eos_token_id=int:1 > /dev/null 2>&1 | grep eval

Output behind this pop-up (a little messy, maybe should have stopped stdout but whatever it's readable):

$ llama-cli -c 200 -n 50 --prompt '<|userprompt|>Hi! How is it going? What’s up?<|endofuserprompt|><|response|>' --model dots1-instruct-q4.gguf --ignore-eos -no-cnv --log-verbose --override-kv tokenizer.ggml.eos_token_id=int:1 > /dev/null 2>&1 | grep eval
Hi! How is it going? What’s up?eval: [ '<|userprompt|>':151646, 'Hi':13048, '!':0, ' How':2585, ' is':374, ' it':432, ' going':2087, '?':30, ' What':3555, 's':748, ' up':705, '?':30, '<|endofuserprompt|>':151647, '<|response|>':151648 ]
Helloeval: [ 'Hello':9707 ]
!eval: [ '!':0 ]
 Ieval: [ ' I':358 ]
’meval: [ 'm':4249 ]
 doingeval: [ ' doing':3730 ]
 welleval: [ ' well':1632 ]
,eval: [ ',':11 ]
 thankeval: [ ' thank':9702 ]
 youeval: [ ' you':498 ]
 foreval: [ ' for':369 ]
 askingeval: [ ' asking':10161 ]
.eval: [ '.':13 ]
 Howeval: [ ' How':2585 ]
 abouteval: [ ' about':911 ]
 youeval: [ ' you':498 ]
?eval: [ '?':30 ]
 Whateval: [ ' What':3555 ]
’seval: [ 's':748 ]
 oneval: [ ' on':389 ]
 youreval: [ ' your':697 ]
 mindeval: [ ' mind':3971 ]
 todayeval: [ ' today':3351 ]
?eval: [ '?':30 ]
 Leteval: [ ' Let':6771 ]
’seval: [ 's':748 ]
 chateval: [ ' chat':6236 ]
!eval: [ '!':0 ]
eval: [ '<|endofresponse|>':151649 ]
eval: [ '<|endofresponse|>':151649 ]
eval: [ '<|endofresponse|>':151649 ]
<peval: [ '<p':7926 ]
>eval: [ '>':29 ]
Caneval: [ 'Can':6713 ]
 youeval: [ ' you':498 ]
 explaineval: [ ' explain':10339 ]
 theeval: [ ' the':279 ]
 differenceeval: [ ' difference':6672 ]
 betweeneval: [ ' between':1948 ]
 aeval: [ ' a':264 ]
 <eval: [ ' <':366 ]
strongeval: [ 'strong':4519 ]
>eval: [ '>':29 ]
meteval: [ 'met':4059 ]
apheval: [ 'aph':1342 ]
llama_perf_context_print: prompt eval time =     176.07 ms /    14 tokens (   12.58 ms per token,    79.51 tokens per second)
llama_perf_context_print:        eval time =    1465.29 ms /    49 runs   (   29.90 ms per token,    33.44 tokens per second)

---

Checked dissonance between the tokenizer_config.json I was using before and llama.cpp's opinion of them:

$ llama-tokenize --model dots1-instruct-q4.gguf --prompt "<|userprompt|><|endofuserprompt|><|response|><|endofresponse|><|im_end|><|im_start|>" --log-disable
151646 -> '<|userprompt|>'
151647 -> '<|endofuserprompt|>'
151648 -> '<|response|>'
151649 -> '<|endofresponse|>'
151645 -> '<|im_end|>'
151644 -> '<|im_start|>'

And then from HF (https://huggingface.co/rednote-hilab/dots.llm1.inst/blob/main/tokenizer_config.json), marked with X I saw disagree:

"151646" -> I can't find this in tokenizer_config.json
"151647" -> "<|userprompt|>" X
"151648" -> "<|endofuserprompt|>" X
"151649" -> "<|response|>" X
"151645" -> "<|im_end|>"
"151644" -> "<|im_start|>"

The tokens for the prompts etc. are off in tokenizer_config.json (also tokenizer.json) compared to llama.cpp. I guess don't trust those files, because 151649 makes a lot more sense if it was really <|endofresponse|> rather than <|response|>, at least based on empiric testing. Threw me off a bit, ugh.

I think there's something I don't understand about convert_*.py scripts because I kinda thought one of those files (tokenizer.json or tokenizer_config.json) was a source of truth what llama.cpp would consider token names and their purpose, but apparently not, or at least not in the way I thought. I guess I never read the details what happens with vocabulary and the set_vocab() in the Python script exactly, I copied mine from Qwen's and it goes to call sentencepiece thing and then gpt2 thing if that doesn't work. Next time I sit down I'll check what's going on here if not otherwise answered somehow, but gotta go for now and did not have time to study before this comment.

---

Tl;dr; 151649 (<|endofreponse|>) seems to work as a stopping token. Don't trust the tokenizer.json or tokenizer_config.json files in HF maybe that say it is <|response|>. --override-kv tokenizer.ggml.eos_token_id=int:151649.

[jukofyork]

Nice work! It might be worth bringing up the messed up config files in the repos' discussions, as it wouldn't be the first time an official uploader of a model has copied the wrong config files!

[Noeda]

Nice work! It might be worth bringing up the messed up config files in the repos' discussions, as it wouldn't be the first time an official uploader of a model has copied the wrong config files!

Yeah, I've kinda come to assume the HF files have some amount of mess-ups by now seeing broken model releases many times by now 😅. I feel confident it is wrong on the HF side, I'll open discussion there when I get to have some focused time again.

Edit: opened discussion here: https://huggingface.co/rednote-hilab/dots.llm1.inst/discussions/4

[ddh0]

For anyone working on implementing the dots.llm1 architecture, this line from the paper might be worth pointing out. Not sure, but just posting it here for visibility:

For each token, the router selects the top-6 isolated experts in addition to the 2 shared experts, resulting in 8 active experts per token. Notably, we employ FP32 precision for the gating layer computations rather than BF16 to ensure
numerical stability and more accurate expert selection during the routing process.

[jukofyork]

For anyone working on implementing the dots.llm1 architecture, this line from the paper might be worth pointing out. Not sure, but just posting it here for visibility:

For each token, the router selects the top-6 isolated experts in addition to the 2 shared experts, resulting in 8 active experts per token. Notably, we employ FP32 precision for the gating layer computations rather than BF16 to ensure
numerical stability and more accurate expert selection during the routing process.

They are always kept as F32 AFAIK:

                # Conditions should closely match those in llama_model_quantize_internal in llama.cpp
                # Some tensor types are always in float32
                if data_qtype is False and (
                    any(
                        self.match_model_tensor_name(new_name, key, bid)
                        for key in (
                            gguf.MODEL_TENSOR.FFN_GATE_INP,
                            gguf.MODEL_TENSOR.POS_EMBD,
                            gguf.MODEL_TENSOR.TOKEN_TYPES,
                            gguf.MODEL_TENSOR.SSM_CONV1D,
                            gguf.MODEL_TENSOR.TIME_MIX_FIRST,
                            gguf.MODEL_TENSOR.TIME_MIX_W1,
                            gguf.MODEL_TENSOR.TIME_MIX_W2,
                            gguf.MODEL_TENSOR.TIME_MIX_DECAY_W1,
                            gguf.MODEL_TENSOR.TIME_MIX_DECAY_W2,
                            gguf.MODEL_TENSOR.TIME_MIX_LERP_FUSED,
                            gguf.MODEL_TENSOR.POSNET_NORM1,
                            gguf.MODEL_TENSOR.POSNET_NORM2,
                            gguf.MODEL_TENSOR.V_ENC_EMBD_POS,
                            gguf.MODEL_TENSOR.A_ENC_EMBD_POS,
                        )
                    )
                    or not new_name.endswith(".weight")
                ):
                    data_qtype = gguf.GGMLQuantizationType.F32

(FFN_GATE_INP is the expert routing tensor)

[ddh0]

The tensor is kept in F32, but does that mean that the gating layer computation is also in F32? (it might, I don't know)

[jukofyork]

The tensor is kept in F32, but does that mean that the gating layer computation is also in F32? (it might, I don't know)

Yeah, I think llama.cpp always used F32 for the activations (there is the GGML_CUDA_F16 option, but I'm not sure it's actually used).

[jacekpoplawski]

they corrected tokenizer.json, tokenizer_config.json and generation_config.json