Strange interaction for DeepZero via text-generation-webui?

[cal066]

I'm trying to integrate AutoAWQ into webui: oobabooga/text-generation-webui#3999
Unfortunately it is causing problems if DeepZero is used, I can't tell where the problem lies.

text-generation-webui  | 2023-09-19 13:13:18 INFO:Loading TheBloke_vicuna-13B-v1.5-16K-AWQ...
text-generation-webui  | [2023-09-19 13:13:19,051] [INFO] [partition_parameters.py:347:__exit__] finished initializing model - num_params = 1, num_elems = 0.16B
text-generation-webui  | 2023-09-19 13:13:19 ERROR:Failed to load the model.
text-generation-webui  | Traceback (most recent call last):
text-generation-webui  |   File "/app/modules/ui_model_menu.py", line 194, in load_model_wrapper
text-generation-webui  |     shared.model, shared.tokenizer = load_model(shared.model_name, loader)
text-generation-webui  |   File "/app/modules/models.py", line 78, in load_model
text-generation-webui  |     output = load_func_map[loader](model_name)
text-generation-webui  |   File "/app/modules/models.py", line 285, in AutoAWQ_loader
text-generation-webui  |     return AutoAWQForCausalLM.from_quantized(quant_path=Path(f'{shared.args.model_dir}/{model_name}'),
text-generation-webui  |   File "/venv/lib/python3.10/site-packages/awq/models/auto.py", line 44, in from_quantized
text-generation-webui  |     return AWQ_CAUSAL_LM_MODEL_MAP[model_type].from_quantized(
text-generation-webui  |   File "/venv/lib/python3.10/site-packages/awq/models/base.py", line 338, in from_quantized
text-generation-webui  |     model = AutoModelForCausalLM.from_config(config=config, torch_dtype=torch_dtype, trust_remote_code=trust_remote_code)
text-generation-webui  |   File "/venv/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 446, in from_config
text-generation-webui  |     return model_class._from_config(config, **kwargs)
text-generation-webui  |   File "/venv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 1168, in _from_config
text-generation-webui  |     model = cls(config, **kwargs)
text-generation-webui  |   File "/venv/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 458, in wrapper
text-generation-webui  |     f(module, *args, **kwargs)
text-generation-webui  |   File "/venv/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 747, in __init__
text-generation-webui  |     self.model = LlamaModel(config)
text-generation-webui  |   File "/venv/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 458, in wrapper
text-generation-webui  |     f(module, *args, **kwargs)
text-generation-webui  |   File "/venv/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 576, in __init__
text-generation-webui  |     self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)
text-generation-webui  |   File "/venv/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 465, in wrapper
text-generation-webui  |     self._post_init_method(module)
text-generation-webui  |   File "/venv/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 984, in _post_init_method
text-generation-webui  |     param.data = param.data.to(self.local_device)
text-generation-webui  | NotImplementedError: Cannot copy out of meta tensor; no data!

Any ideas?

[casper-hansen]

I’m not quite sure. I have seen this error before, but I am not sure what is causing it to happen.

Did you compile from main or did you use the pypi package?

[cal066]

This is from main, git revision 8788fe1. I pointed pip to github.

[casper-hansen]

I think I understand better now. DeepSpeed is trying to copy data that does not exist. This happens because we start by initializing the model with empty weights:

AutoAWQ/awq/models/base.py

Lines 337 to 338 in 8788fe1

	with init_empty_weights():
	model = AutoModelForCausalLM.from_config(config=config, torch_dtype=torch_dtype, trust_remote_code=trust_remote_code)

In general, I am not sure AutoAWQ is even compatible with DeepSpeed since we rely on accelerate.
huggingface/accelerate#1530 (comment)

[cal066]

That was what I suspected when I saw some comments on DeepSpeed about copying empty weights. Hopefully there is a way to stop DeepSpeed from kicking in for model types that are incompatible.

[casper-hansen]

Closing this since we use accelerate.