Skip to content

[Bug][Qwen3.5-4B][Qwen3.6-35B-A3B] Accuracy=0 #1836

@XuehaoSun

Description

@XuehaoSun

Problem Description

|    Tasks     |Version|Filter|n-shot|  Metric  |   |Value|   |Stderr|
|--------------|------:|------|-----:|----------|---|----:|---|-----:|
|lambada_openai|      1|none  |     0|acc       |↑  |    0|±  |     0|
|              |       |none  |     0|perplexity|↓  |  NaN|±  |   NaN|

Qwen3.6-35B-A3B: log
Qwen3.5-4B: log

Reproduction Steps

auto-round --model_name /models/Qwen3.5-4B --bits 4 --iters 0 --tasks lambada_openai
auto-round --model_name /models/Qwen3.6-35B-A3B --bits 4 --iters 0 --tasks lambada_openai

Environment Information

No response

Error Logs

Additional Context

No response

Metadata

Metadata

Assignees

Labels

Type

No fields configured for Bug.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions