[BUG] saving problem for larger model which require shard checkpoints

[ChenMnZ]

Hi,

I think this bug is similar to #252.

I found that the following code can load any GPTQ v2 models successfully:

model = GPTQModel.from_quantized(args.model, device_map='auto',torch_dtype=torch.float16)

However, when I try to saving like：

model.save_pretrained(args.save_dir)  

I found that no any checkpoints is saved. Additionally, if I try to save as:

model.model.save_pretrained(args.save_dir)  

The shard checkpoints can be saving. However, the output results of model are wrong when reload the saved checkpoints throug model.model.save_pretrained.

Overall, model.save_pretrained cannot save shard checkpoints. model.model.save_pretrained can save shard checkpoints while produce wrong outputs.

[Qubitium]

@ChenMnZ We will check to see the save_pretrained is incorrectly overridded by GPTQModel. Can you also test directly using .save_quantized() viat the sharded unit sample/test here

https://github.com/ModelCloud/GPTQModel/blob/main/tests/test_sharded.py#L88-L91

model.save_quantized(
                tmp_dir,
                max_shard_size="100MB",
            )

[Qubitium]

fixed in pr #259

[Qubitium]

We have reverted PR #259 in #277 . We will use a more compatible fix in new PR.

In the mean-time, do not call save_quantized() after loading via from_quantized(). There are multiple bugs in this code path if you use special backends. Real fix coming with better backward compat.

The crust of the bug is due to the introduction of backends and bitblas/qbit/vllm/sglang, etc will break save_quantized if loading a already quantized model with these special backends. from_quantized api is meant for inference, for not re-sharding or quantlinear packing in save_quantized. We need to allow safe packing for those backends we support after loading a pre-qunated model and for those usafe backends, we need throw error.

[Qubitium]

fixed in Pr #296.