No speedup using semi-structured sparsity

[BDHU]

When I convert linear layer weights with to_sparse_semi_structured(module.weight), there doesn't appear to be any speedup when I set the dimension of the linear layer as 4096x4096 with input size being 1x4096. Pytorch is compiled with cuSparseLT. Is there any reason why there is no speedup? In addition, the sparse alternative consumes more memory than the dense one. Any hint will be helpful!

cc @alexsamardzic @nikitaved @pearu @cpuhrsch @amjames @bhosmer

[cpuhrsch]

cc @jcaip

[jcaip]

Hi @BDHU 4096x4096 @ 4096x1 is too small a matrix size to show speedups.

Semi-structured sparsity is not faster in all cases over dense matmul, depending on the shapes of the matmul shapes.

Running the quickstart section but adjusted for your shapes I see

Dense: 0.022ms Sparse: 0.196ms | Speedup: 0.111x

You need larger weight matrix shapes to show the effectiveness of semi-structured sparsity.

As for the increased memory consumption, that's because 1x4096 is not technically a supported shape - so we pad to the nearest multiple of 64/32/16 depending on dtype. For float16, it's 32 so that 1x4096 gets padded to a 32x4096 matrix and then we slice the results so what you see is the correct shape. (This contributes to the less than ideal speedup)