unexpected slow speed on windonws with RTX4060

[kalle07]

from auto_round import AutoRound
from pathlib import Path

model_dir = Path(r"c:\Users\XXX\Documents\python\autoround\facebook")  # <-- hier deinen Pfad eintragen

# --- Überprüfen, ob der Pfad existiert und ein Verzeichnis ist ---
if not model_dir.exists():
    print(f"❌ Der Pfad '{model_dir}' existiert nicht.")
elif not model_dir.is_dir():
    print(f"⚠️ Der Pfad '{model_dir}' ist keine Verzeichnisstruktur.")
else:
    print(f"📁 Inhalte des Verzeichnisses: {model_dir}\n")

    # --- Alle Dateien im Verzeichnis auflisten ---
    files = [f for f in model_dir.iterdir() if f.is_file()]

    if not files:
        print("ℹ️ Keine Dateien im Verzeichnis gefunden.")
    else:
        for file in files:
            print(f" - {file.name}")

# Available schemes: "W2A16", "W3A16", "W4A16", "W8A16", "NVFP4", "MXFP4" (no real kernels), "GGUF:Q4_K_M", etc.
ar = AutoRound(model_dir, scheme="W4A16", iters=50, lr=5e-3)

# Highest accuracy (4–5× slower).
# `low_gpu_mem_usage=True` saves ~20GB VRAM but runs ~30% slower.
# ar = AutoRound(model_name_or_path, nsamples=512, iters=1000, low_gpu_mem_usage=True)

# Faster quantization (2–3× speedup) with slight accuracy drop at W4G128.
# ar = AutoRound(model_name_or_path, nsamples=128, iters=50, lr=5e-3)

# Supported formats: "auto_round" (default), "auto_gptq", "auto_awq", "llm_compressor", "gguf:q4_k_m", etc.
ar.quantize_and_save(output_dir=r"c:\Users\XXX\Documents\python\autoround\tmp_autoround", format="q4_k_m")

thats shows the shell:

2025-10-08 16:52:23,670 INFO utils.py L164: NumExpr defaulting to 16 threads.
📁 Inhalte des Verzeichnisses: c:\Users\XXX\Documents\python\autoround\facebook

• config.json
• flax_model.msgpack
• generation_config.json
• gitattributes
• LICENSE.md
• merges.txt
• pytorch_model.bin
• README.md
• special_tokens_map.json
• tf_model.h5
• tokenizer_config.json
• vocab.json
  Traceback (most recent call last):
  File "C:\Users\XXX\Documents\python\autoround\auto01.py", line 24, in
  ar = AutoRound(model_dir, scheme="W4A16", iters=50, lr=5e-3)
  TypeError: AutoRound.init() missing 1 required positional argument: 'tokenizer'

what do i miss ?

[wenhuach21]

it seems you are using an old version of AutoRound, please use the latest version (>=0.7.0) or follow the legacy usage to pass the tokenizer manually

from transformers import AutoModelForCausalLM, AutoTokenizer
from auto_round import AutoRound

model_name = "Qwen/Qwen3-0.6B"
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)

bits, group_size, sym = 4, 128, True
autoround = AutoRound(model, tokenizer, bits=bits, group_size=group_size, sym=sym)

## the best accuracy, 4-5X slower, low_gpu_mem_usage could save ~20G but ~30% slower
# autoround = AutoRound(model, tokenizer, nsamples=512, iters=1000, low_gpu_mem_usage=True, bits=bits, group_size=group_size, sym=sym)

## 2-3X speedup, slight accuracy drop at W4G128
# autoround = AutoRound(model, tokenizer, nsamples=128, iters=50, lr=5e-3, bits=bits, group_size=group_size, sym=sym )

output_dir = "./tmp_autoround"
## format= 'auto_round'(default), 'auto_gptq', 'auto_awq'
autoround.quantize_and_save(output_dir, format='auto_round') 

[kalle07]

okay seems python 3.10 load the old version, iam one step closer use python 3.12, but

...
torch_dtype is deprecated! Use dtype instead!
2025-10-08 18:16:45 WARNING autoround.py L2533: amp is set to FALSE as the current cpu device does not support the 'bf16' data type.
2025-10-08 18:16:45 INFO autoround.py L271: using torch.float32 for quantization tuning
2025-10-08 18:16:45 INFO autoround.py L276: 'enable_torch_compile' is set to False by default. Enabling it can reduce tuning cost by 20%, but it might throw an exception.
2025-10-08 18:16:45 INFO autoround.py L1294: start to cache block inputs
2025-10-08 18:16:45,605 INFO config.py L54: PyTorch version 2.8.0 available.
2025-10-08 18:16:50 INFO autoround.py L1308: caching done
Quantizing model.decoder.layers.0: 0%| | 0/12 [00:00<?, ?it/s]

all CPU cores run (even if i have an 4060RTX) and nothing happend 10min
its that facebook model 250MB

thx for hints

[wenhuach21]

please check your installtion of torch. Based on the log，cuda device has not been found and your cpu does not support bf16/fp16.
2025-10-08 18:16:45 WARNING autoround.py L2533: amp is set to FALSE as the current cpu device does not support the 'bf16' data type

[kalle07]

okay thx , works at least for "auto_round"!!!

maybe you write a hint that is extra to install (system depend)
for me worked (torch 2.8.0+cu126):
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126
(3GB)

for gguf it apperas an error:
change line:
autoround.quantize_and_save(output_dir, format='gguf:q4_k_m')

File "c:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round\autoround.py", line 659, in quantize_and_save
format_list = self.parse_format_to_list(format)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round\autoround.py", line 518, in parse_format_to_list
_gguf_args_check(self, format, model_type=ModelType.TEXT)
File "c:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round\utils.py", line 1197, in _gguf_args_check
raise ImportError(
ImportError: Please use the latest gguf-py for gguf:q4_k_m, you can use the following command to install it:
git clone https://github.com/ggml-org/llama.cpp.git && cd llama.cpp/gguf-py && pip install .

git clone https://github.com/ggml-org/llama.cpp.git
and I installed requirements...
i done this... but same error ... Is that independent of your work?

sry for that questions i think iam not that bad ?!? ;)

[wenhuach21]

gguf does not support opt-125m, you could switch to qwen or other popular models.

[kalle07]

sry ... next step (thx for help)

i see ... now it tried it with JAN (qwen based)
https://huggingface.co/janhq/Jan-v1-2509
4B model (8GB)

i used low-mem (i have 16GB Vram)

--- AutoRound initialisieren ---

bits, group_size, sym = 4, 128, True
autoround = AutoRound(model, tokenizer, bits=bits, group_size=group_size, sym=sym, low_gpu_mem_usage=True)
autoround.quantize_and_save(output_dir, format='auto_round')

(venv) c:\Users\xxx\Documents\python\autoround>py auto03.py
📁 Inhalte des Verzeichnisses: c:\Users\xxx\Documents\python\autoround\Jan-v1-2509

• added_tokens.json
• chat_template.jinja
• config.json
• generation_config.json
• gitattributes
• merges.txt
• model-00001-of-00002.safetensors
• model-00002-of-00002.safetensors
• model.safetensors.index.json
• README.md
• special_tokens_map.json
• tokenizer.json
• tokenizer_config.json
• trainer_state.json
• training_args.bin
• vocab.json
  torch_dtype is deprecated! Use dtype instead!
  Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 32.74it/s]
  2025-10-09 10:58:56 INFO autoround.py L271: using torch.bfloat16 for quantization tuning
  2025-10-09 10:58:56 INFO autoround.py L276: 'enable_torch_compile' is set to False by default. Enabling it can reduce tuning cost by 20%, but it might throw an exception.
  2025-10-09 10:58:56 INFO autoround.py L1294: start to cache block inputs
  2025-10-09 10:58:58,554 INFO config.py L54: PyTorch version 2.8.0+cu126 available.
  2025-10-09 10:59:05 INFO autoround.py L1308: caching done
  Quantizing model.layers.0: 0%| | 0/36 [00:00<?, ?it/s]T

---

GPU memory: first it used 8GB after 5sec 12 and after 5 sec 16 ... GPU runs 100% but no temperature rise-up and nothing happend

[wenhuach21]

Is it hanging? How long have you waited? Please try setting iters=1 for a quick test to check whether it’s actually hanging or just running slowly.

[wenhuach21]

I did a quick test, at least on my local machine, it works well, the vram is about 10GB

[kalle07]

wooo fast answer ;)

with iters=1 it runs with 20s/it

and my vram looks like this (before only 200MB used out of 16GB)

the short peaks is 16GB

hmmm what could that mean?

[wenhuach21]

ok. it may be related to your gpu and cpu. rm low gpu mem usage ，enable torch compile and set batch-size to 4 or seqlen to 512， which should be faster

[kalle07]

its a usual intel i710700 and a RTX4060 (all Ai software runs fine on my system)

with batch 4 it runs with 13GB and 100s/its and i will see if runs untill the end ;)

with "enable_torch_compile=True"
error:

torch_dtype is deprecated! Use dtype instead!
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 33.00it/s]
2025-10-09 12:30:16 INFO autoround.py L271: using torch.bfloat16 for quantization tuning
2025-10-09 12:30:16 INFO autoround.py L1294: start to cache block inputs
2025-10-09 12:30:18,888 INFO config.py L54: PyTorch version 2.8.0+cu126 available.
2025-10-09 12:30:26 INFO autoround.py L1308: caching done
Quantizing model.layers.0: 0%| | 0/36 [00:00<?, ?it/s]c:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\torch_inductor\compile_fx.py:282: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting torch.set_float32_matmul_precision('high') for better performance.
warnings.warn(
W1009 12:30:28.874000 12844 venv\Lib\site-packages\torch_inductor\utils.py:1436] [3/0] Not enough SMs to use max_autotune_gemm mode
Traceback (most recent call last):
File "c:\Users\xxx\Documents\python\autoround\auto03.py", line 41, in
autoround.quantize_and_save(output_dir, format='auto_round')
File "c:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round\autoround.py", line 669, in quantize_and_save
model, _ = self.quantize()
^^^^^^^^^^^^^^^
File "c:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round\autoround.py", line 1335, in quantize
self.quant_blocks(
File "c:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round\autoround.py", line 2366, in quant_blocks
q_input, input_ids = quant_block(
^^^^^^^^^^^^
File "c:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\torch_dynamo\eval_frame.py", line 749, in compile_wrapper
raise e.remove_dynamo_frames() from None # see TORCHDYNAMO_VERBOSE=1
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\torch_inductor\scheduler.py", line 4042, in create_backend
raise TritonMissing(inspect.currentframe())
torch._inductor.exc.TritonMissing: Cannot find a working triton installation. Either the package is not installed or it is too old. More information on installing Triton can be found at: https://github.com/triton-lang/triton

Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"

---

i remember triton in windows dont work ?!?

[wenhuach21]

oh, I am not aware of this information as I have never run auto-round on windows.

025-10-09 07:45:22 INFO autoround.py L2853: quantized 7/7 layers in the block, loss iter 0: 0.001491 -> iter 161: 0.000602
Quantizing model.layers.5:  14%|██████████████████████████████▉                                                                                                                                                                                                | 5/36 [01:20<07:11, 13.93s/it]2025-10-09 07:45:34 INFO autoround.py L2853: quantized 7/7 layers in the block, loss iter 0: 0.002287 -> iter 168: 0.001017
Quantizing model.layers.6:  17%|█████████████████████████████████████▏                                                                                                                                                                                         | 6/36 [01:32<06:38, 13.30s/it]2025-10-09 07:45:47 INFO autoround.py L2853: quantized 7/7 layers in the block, loss iter 0: 0.044403 -> iter 166: 0.002818
Quantizing model.layers.7:  19%|███████████████████████████████████████████▎                                                                                                                                                                                   | 7/36 [01:44<06:13, 1

Then just set the other two hyperparameters or combine them. For extreme large model, you could set iters to 0 to enable rtn model, which may suffer notable accuracy drop

[wenhuach21]

@n1ck-guo Please take a look when you have some bandwidth and if it is possible for us to borrow such devices.