You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In addition, looking at the GPU usage, it seems that it has become a parallel pipeline. I use four cores to load the model, but only one GPU utilization is 100% at the same time.
Have you encountered the same problem as me again? Is the inference speed of your quantized model normal?
The text was updated successfully, but these errors were encountered:
Hi @ACoder-J !
The inference speed is really bottlenecked by transformers for multiple reasons (transformers being slow and kernel launches taking long).
One way to achieve proper speedup is to compile your model into a cuda graph. It's a new feature added to PyTorch in 2.0.0 and refined in 2.2.1. People of transformers have also been working hard to properly support and document it. To check out what you can do with it already, please refer to this AQLM demo.
Another way to properly serve AQLM is to use the vllm integration. It's not complete yet but it might be usable already (not sure).
Why is it that after I quantize, my inference is 2 times slower than the original `model?
Quantize command:python main.py $MODEL_PATH $DATASET_PATH --nsamples=1024 --num_codebooks=1 --nbits_per_codebook=16 --in_group_size=8 --relative_mse_tolerance=0.01 --finetune_relative_mse_tolerance=0.001 --finetune_batch_size=32 --local_batch_size=1 --offload_activations --wandb --save $SAVE_PATH --dtype float32
Quantitative results:
![image](https://private-user-images.githubusercontent.com/11597706/316371684-1bc41c17-8626-4b9c-b5ba-87f7734c898f.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUwMTc2OTgsIm5iZiI6MTcyNTAxNzM5OCwicGF0aCI6Ii8xMTU5NzcwNi8zMTYzNzE2ODQtMWJjNDFjMTctODYyNi00YjljLWI1YmEtODdmNzczNGM4OThmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MzAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODMwVDExMjk1OFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTBlZTgwOTYxODc0MjQwMjFhMGJmMjk0MzNmNTZiMmEzNmRiZjAzOWRjYmY5MzRkMmQwNGQzNDM1OGRjZDg5YmImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.QjDVoYX6GmWeKL_DabPpU5562sLTMPik6hr0rpoq1rY)
In addition, looking at the GPU usage, it seems that it has become a parallel pipeline. I use four cores to load the model, but only one GPU utilization is 100% at the same time.
![image](https://private-user-images.githubusercontent.com/11597706/316371532-87ed8f71-d98c-464c-8568-4d0f101f6962.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUwMTc2OTgsIm5iZiI6MTcyNTAxNzM5OCwicGF0aCI6Ii8xMTU5NzcwNi8zMTYzNzE1MzItODdlZDhmNzEtZDk4Yy00NjRjLTg1NjgtNGQwZjEwMWY2OTYyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MzAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODMwVDExMjk1OFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWFjOTQ4ZDczYjJkZDVjNWNmOGJlNTdhZTk4NTk1OTRmMTkzYmQxNDMxMmUwZmE5MjY4NzY1MjU3NTA5ZDlkODEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.wGCW1uxmSHGWst4Ccco-fT_bI_qVoQaOgtDLhYOeenU)
Have you encountered the same problem as me again? Is the inference speed of your quantized model normal?
The text was updated successfully, but these errors were encountered: