❓ [Question] No improvement when I use sparse-weights? #1340
Description
❓ Question
No speed improvement when I use sparse-weights.
I just modified this notebook https://github.com/pytorch/TensorRT/blob/master/notebooks/Hugging-Face-BERT.ipynb
And add the sparse_weights=True in the compile part. I also changed the regional bert-base model when I apply 2:4 sparse on most parts of the FC layers.
But whether I set the "sparse_weights=True", the results look like no changes.
Here are some results.
set sparse_weights=False
set sparse_weights=True
Environment
Build information about Torch-TensorRT can be found by turning on debug messages
- PyTorch Version (e.g., 1.0): 1.13
- CPU Architecture:x86-64
- OS (e.g., Linux):Ubuntu 18.04
- How you installed PyTorch (
conda,pip,libtorch, source): - Build command you used (if compiling from source):
- Are you using local sources or building from archives:
- Python version: 3.8
- CUDA version: 11.7.1
- GPU models and configuration: Nvidia A100 GPU & CUDA Driver Version 515.65.01
- Any other relevant information:
Additional context
