convert.py - RuntimeError: CUDA error: invalid configuration argument

[Thireus]

python convert.py -i ~/safetensor/model -o ~/EXL2/model_4bit -c ~/EXL2/0000.parquet -b 4.0 -hb 6 -l 4096 -ml 4096

 -- Beginning new job
 -- Input: /home/user/safetensor/model
 -- Output: /home/user/EXL2/model_4bit
 -- Calibration dataset: /home/user/EXL2/0000.parquet, 100 / 16 (16) rows, 4096 tokens per sample
 -- Target bits per weight: 4.0 (decoder), 6 (head)
 -- Tokenizing samples (measurement)...
/home/user/exllamav2/conversion/tokenize.py:16: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  tokens = tokenizer.encode(row[0])
 -- Token embeddings (measurement)...
 -- Measuring quantization impact...
 -- Layer: model.layers.0 (Attention)
Traceback (most recent call last):
  File "/home/user/exllamav2/convert.py", line 168, in <module>
    measure_quant(job, save_job, model)
  File "/home/user/exllamav2/conversion/quantize.py", line 184, in measure_quant
    outputs = module.forward(x, cache, attn_mask, intermediates = True)
  File "/home/user/exllamav2/exllamav2/attn.py", line 195, in forward
    return self.forward_torch(hidden_states, cache, attn_mask, past_len, intermediates)
  File "/home/user/exllamav2/exllamav2/attn.py", line 460, in forward_torch
    key_states = self.repeat_kv(key_states, self.model.config.num_key_value_groups)
  File "/home/user/exllamav2/exllamav2/attn.py", line 188, in repeat_kv
    hidden_states = hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)
RuntimeError: CUDA error: invalid configuration argument
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Any idea what could be the cause of this issue?

Edit: Bad or unsupported safetensor files it seems
Edit2: That wasn't bad safetensors in the end, cf TL;DR

---

TL;DR: The issue is likely due to VRAM limitation, 24GB not enough for -ml 4096 -l 4096 at the moment. Some VRAM otpimization code is underway, until then you must use the default -ml 2048 -l 2048.

[DocShotgun]

How exactly did you manage to resolve this? Getting this error right now.

I simply loaded my model and saved it in safetensors using transformers with model.save_pretrained() and safe_serialization = True. Not sure why the safetensor files would be bad after that.

[Thireus]

I haven't resolved it yet due to lack of time.

So far I've used https://github.com/oobabooga/text-generation-webui/blob/main/convert-to-safetensors.py on 70B models which led to the issue and realized that it's a safetensors conversion problem.

I've asked around from people who successfully converted to good safetensors, and I'm now trying their custom scripts:

• https://huggingface.co/Panchovix/airoboros-l2-70b-gpt4-1.4.1-safetensors/blob/main/bin2safetensors/convert.py (https://huggingface.co/Panchovix/airoboros-l2-70b-gpt4-1.4.1_5.0bpw-h6-exl2/discussions/1#65023208a33ce3ae285b8924)
• https://gist.github.com/epicfilemcnulty/1f55fd96b08f8d4d6693293e37b4c55e (Conversion help #25 (comment))

The scripts above should work. I'm currently in the middle of converting with epicfilemcnulty's script.

The issue could be elsewhere indeed. (I'm also thinking float16-related maybe?)

Edit: Sadly, I'm facing the same issue with epicfilemcnulty's script.
Edit2: Same issue with Panchovix's convert.py script.

[turboderp]

There are a couple of issues currently with BF16 models that should be resolved shortly. This could also be a VRAM issue. Torch (especially with extensions that maybe don't do all the error handling they should be doing) can sometimes end up in an invalid state rather than crashing with an out-of-memory error, and due to the asynchronous nature of CUDA the function that triggers an exception is often not the one that's actually failing.

First, make sure you're not running out of memory. Run nvtop or some such in a separate terminal, check if you're running low on memory. Also, you don't have to calibrate on the full sequence length of the model. I find 2048 tokens works well enough even for a 4k-context model.

The implementation assumes that all tensors are contiguous when loaded from the .safetensors file(s), so that's something to be mindful of.

And while it should handle FP32 weights when loading an uncoverted model, they will use a lot more memory, so that's another possible source of OoM issues. Is it possible the difference between the "good" and "bad" .safetensors files in this case is that the good ones are already downcast to FP16?

[Thireus]

I see, thank you for the additional tips @turboderp, I'm also strongly guessing BF16-related issue - I'll have a look at VRAM usage, I'm using a 3090, so I believe VRAM shouldn't be the issue.

• Successful results with CodeLlama-13B-Instruct-fp16 - https://huggingface.co/TheBloke/CodeLlama-13B-Instruct-fp16
• Unsuccessful results when converting https://huggingface.co/WizardLM/WizardLM-70B-V1.0 to safetensors, also tried Float16 from https://huggingface.co/simsim314/WizardLM-70B-V1.0-HF to safetensors.

Also ruled out contiguous issue, as I've used this script which ensures tensors are contiguous:

• https://gist.github.com/epicfilemcnulty/1f55fd96b08f8d4d6693293e37b4c55e#file-2safetensors-py-L45-L46

[Thireus]

@turboderp, I can confirm this isn't a VRAM issue.

I have also tested your latest commit (5ccd29c). I saw you made some changes to print errors when safetensor files don't have the expected data. But I'm still getting the same error message without further information.

I don't think it should matter, but I'm converting from .bin to .safetensor using a separate machine with cpu only.

[turboderp]

Are you also getting the deprecation warning at the start? I'm starting to think that may have something to do with it. There's another issue I'm working on where that also shows up, so it could have something to do with the calibration data being loaded incorrectly.

[Thireus]

@turboderp, I can confirm this isn't a VRAM issue.

I have also tested your latest commit (5ccd29c). I saw you made some changes to print errors when safetensor files don't have the expected data. But I'm still getting the same error message without further information.

[Thireus]

@turboderp, yes the depreciation warning is now present indeed.

[DocShotgun]

Some additional details from my end:

1. I also encountered this error while attempting to quant a 70b model, while I previously quanted 13b models without issue.
2. The environment was an L4 runpod instance with the base torch 2.0.1 docker and python 3.10. It has 250gb ram and 48gb vram, which should be adequate for this.
3. The error occurs almost immediately when attempting to do measurement on attention layer 0.

[turboderp]

Do you get the same error without the -l 4096 -ml 4096. Cause that really shouldn't be needed, and while it shouldn't matter with 48 GB of VRAM, it does still use up memory.

Also, I'm wondering if the Parquet file you're using contains enough data for 409600 tokens. I know at least there's a wikitext-test split floating around with some 600 kB of text. If that runs out, I'm not sure what would happen.

[DocShotgun]

I ended up killing the instance before trying without the 4096 calibration context, so unsure on that one lol.

But I used the same parquet with several megabytes of data that I used for the 13b quants at 4096 calibration, so there should be enough data there. It seemed to get past the tokenization of the dataset properly, as it’s after it says it’s going to work on attention layer 0 that I errored out.

[Thireus]

Happy to report that with your latest commit and without using "-ml 4096", the issue is gone.

python convert.py -i ~/safetensor/model -o ~/EXL2/model_4bit -c ~/EXL2/0000.parquet -b 4.0 -hb 6 -l 4096
 -- Beginning new job
 -- Input: /home/user/safetensor/model
 -- Output: /home/user/EXL2/model_4bit
 -- Calibration dataset: /home/user/EXL2/0000.parquet, 100 / 16 (16) rows, 4096 tokens per sample
 -- Target bits per weight: 4.0 (decoder), 6 (head)
 -- Tokenizing samples (measurement)...
 -- Token embeddings (measurement)...
 -- Measuring quantization impact...
 -- Layer: model.layers.0 (Attention)
 -- Linear: model.layers.0.self_attn.q_proj
 -- 0.05:3b/0.95:2b 32g s4         2.17 bpw    rfn_error: 0.00967
 -- 0.25:3b/0.75:2b 32g s4         2.38 bpw    rfn_error: 0.00922
 -- 0.1:4b/0.4:3b/0.5:2b 32g s4    2.72 bpw    rfn_error: 0.00573
 -- 0.1:4b/0.9:3b 32g s4           3.22 bpw    rfn_error: 0.00560
 -- 1.0:3b 128g s4                 3.03 bpw    rfn_error: 0.01691
 -- 1.0:3b 32g s4                  3.13 bpw    rfn_error: 0.00896
 -- 0.05:4b/0.95:3b 32g s4         3.17 bpw    rfn_error: 0.00566

But most importantly!!!! The output directory needs to be empty first. rm -rf ~/EXL2/model_4bit && mkdir ~/EXL2/model_4bit, otherwise for some reason it will attempt to reuse "-ml 4096" again (this time from the saved job.json file) even though it's no longer mentioned as a parameter of the command. That behaviour made troubleshooting this issue quite hard.

[Thireus]

Well, after 22h of quantization, that happened:

...
 -- 0.4:6b/0.6:5b 32g s4           5.53 bpw    rfn_error: 0.01424
 -- 1.0:6b 128g s4                 6.03 bpw    rfn_error: 0.00961
 -- 0.1:8b/0.9:6b 128g s4          6.22 bpw    rfn_error: 0.00936
 -- 1.0:8b 32g s4                  8.13 bpw    rfn_error: 0.00747
 -- Time: 14.58 seconds
 -- Linear: model.layers.79.mlp.down_proj
 -- 0.05:3b/0.95:2b 32g s4         2.17 bpw    rfn_error: 0.04251
 -- 0.25:3b/0.75:2b 32g s4         2.38 bpw    rfn_error: 0.03793
 -- 0.1:4b/0.4:3b/0.5:2b 32g s4    2.72 bpw    rfn_error: 0.03168
 -- 0.1:4b/0.9:3b 32g s4           3.22 bpw    rfn_error: 0.01941
 -- 1.0:3b 128g s4                 3.03 bpw    rfn_error: 0.02422
 -- 1.0:3b 32g s4                  3.13 bpw    rfn_error: 0.02196
 -- 0.05:4b/0.95:3b 32g s4         3.17 bpw    rfn_error: 0.02002
 -- 0.4:4b/0.6:3b 32g s4           3.53 bpw    rfn_error: 0.01706
 -- 0.6:4b/0.4:3b 64g s4           3.66 bpw    rfn_error: 0.01630
 -- 1.0:4b 128g s4                 4.03 bpw    rfn_error: 0.01259
 -- 1.0:4b 32g s4                  4.13 bpw    rfn_error: 0.01116
 -- 1.0:5b 128g s4                 5.03 bpw    rfn_error: 0.00671
 -- 0.1:6b/0.9:5b 32g s4           5.22 bpw    rfn_error: 0.00626
 -- 0.4:6b/0.6:5b 32g s4           5.53 bpw    rfn_error: 0.00590
 -- 1.0:6b 128g s4                 6.03 bpw    rfn_error: 0.00430
 -- 0.1:8b/0.9:6b 128g s4          6.23 bpw    rfn_error: 0.00397
 -- 1.0:8b 32g s4                  8.13 bpw    rfn_error: 0.00455
 -- Time: 742.49 seconds
 -- Layer: model.norm (RMSNorm)
 -- Layer: lm_head (Linear)
 -- Calibration perplexity (base): 5.3018
 -- Optimizing...
 -- rfn_error: 0.27881  bpw: 2.17262
 -- rfn_error: 0.13940  bpw: 2.84866
 -- rfn_error: 0.06970  bpw: 3.74662
 -- rfn_error: 0.03485  bpw: 4.77840
 -- rfn_error: 0.05228  bpw: 4.27941
 -- rfn_error: 0.06099  bpw: 3.90092
 -- rfn_error: 0.05663  bpw: 4.02832
 -- rfn_error: 0.05881  bpw: 3.96083
 -- rfn_error: 0.05772  bpw: 3.99127
 -- rfn_error: 0.05718  bpw: 4.00975
 -- rfn_error: 0.05745  bpw: 3.99444
 -- rfn_error: 0.05731  bpw: 3.99950
 -- rfn_error: 0.05731  bpw: 3.99950
 -- rfn_error: 0.05731  bpw: 3.99950
 -- rfn_error: 0.05731  bpw: 3.99950
 -- rfn_error: 0.05731  bpw: 3.99950
 -- rfn_error: 0.05731  bpw: 3.99950
 -- rfn_error: 0.05731  bpw: 3.99950
 -- Tokenizing samples...
 -- Token embeddings again...
 -- Quantizing...
 -- Layer: model.layers.0 (Attention)
Traceback (most recent call last):
  File "/home/user/exllamav2/convert.py", line 195, in <module>
    quant(job, save_job, model)
  File "/home/user/exllamav2/conversion/quantize.py", line 462, in quant
    outputs = module.forward(x, cache, attn_mask, intermediates = True)
  File "/home/user/exllamav2/exllamav2/attn.py", line 195, in forward
    return self.forward_torch(hidden_states, cache, attn_mask, past_len, intermediates)
  File "/home/user/exllamav2/exllamav2/attn.py", line 460, in forward_torch
    key_states = self.repeat_kv(key_states, self.model.config.num_key_value_groups)
  File "/home/user/exllamav2/exllamav2/attn.py", line 188, in repeat_kv
    hidden_states = hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)
RuntimeError: CUDA error: invalid configuration argument
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[turboderp]

It looks like you've run out of memory when moving from the smaller sample of the dataset used for measurement to the full sample used for quantization. I'm working on memory optimizations that should help with that. Expect an update in a little bit.

In the meantime, it seems you've successfully run the measurement pass, so the progress should be saved.

[Thireus]

I see! Indeed the progress is now saved and completed Measuring Quantization (-ml optional parameter was set to default 2048).

I was thinking that now the issue was with the -l parameter which I had set to 4096. So I edited the job.json and set it back to 2048, but that didn't help, still getting the same error.

 -- Resuming job
 !! Note: Overriding options with settings from existing job
 -- Input: /home/user/safetensor/model
 -- Output: /home/user/EXL2/model_4bit
 -- Calibration dataset: /home/user/EXL2/0000.parquet, 100 / 16 (16) rows, 2048 tokens per sample
 -- Target bits per weight: 4.0 (decoder), 6 (head)
 -- Quantizing...
 -- Layer: model.layers.0 (Attention)
Traceback (most recent call last):
  File "/home/user/exllamav2/convert.py", line 195, in <module>
    quant(job, save_job, model)
  File "/home/user/exllamav2/conversion/quantize.py", line 462, in quant
    outputs = module.forward(x, cache, attn_mask, intermediates = True)
  File "/home/user/exllamav2/exllamav2/attn.py", line 195, in forward
    return self.forward_torch(hidden_states, cache, attn_mask, past_len, intermediates)
  File "/home/user/exllamav2/exllamav2/attn.py", line 460, in forward_torch
    key_states = self.repeat_kv(key_states, self.model.config.num_key_value_groups)
  File "/home/user/exllamav2/exllamav2/attn.py", line 188, in repeat_kv
    hidden_states = hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)
RuntimeError: CUDA error: invalid configuration argument
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

When you say memory, do you mean RAM or VRAM? I currently sit with 6GB+128GBswap of free memory and 24GB of VRAM. When I run the job again, it takes 3-4 sec to produce the error, the RAM receives about 2GB and VRAM about 1.5GB before the error happens, so barely filling anything.

I'll try your updated code as soon as it drops, thank you so much for the support!