Add support for converting a GPTQ model to 2:4 structured sparse marlin format for 4-bit and 8-bit

[alexm-neuralmagic]

This diffs adds the ability to convert a GPTQ model to the 2:4 structured sparse marlin format. This diff is intended to serve as reference for the MLE team to integrate their code (and not be landed on autogptq)

[robertgshaw2-neuralmagic]

@AniZpZ here's current hacked up script we have

Note: accuracy will not be great for post-training sparsity 2:4.

We have been investing in making sparse foundational models that have sparsity in them

see here for the research we are expanding up

[AniZpZ]

@AniZpZ here's current hacked up script we have

Note: accuracy will not be great for post-training sparsity 2:4.

We have been investing in making sparse foundational models that have sparsity in them

see here for the research we are expanding up

Much appreciated ! Is there a plan to make your research open source ?
I don't expect a good accuracy for post-training sparsity 2:4. Have you guys looked into other unstructured pruning methods or dynamic pruning methods?

[robertgshaw2-neuralmagic]

@AniZpZ here's current hacked up script we have
Note: accuracy will not be great for post-training sparsity 2:4.
We have been investing in making sparse foundational models that have sparsity in them
see here for the research we are expanding up

Much appreciated ! Is there a plan to make your research open source ? I don't expect a good accuracy for post-training sparsity 2:4. Have you guys looked into other unstructured pruning methods or dynamic pruning methods?

Yes: https://huggingface.co/collections/neuralmagic/sparse-foundational-llama-2-models-65f48cec6396309f02e74d21

• 2:4 models coming soon

We are currently focused on pruning Llama-3 on our cluster

We have done extensive research into pruning with our friends at IST.

@AniZpZ are you on discord?

[AniZpZ]

@AniZpZ here's current hacked up script we have
Note: accuracy will not be great for post-training sparsity 2:4.
We have been investing in making sparse foundational models that have sparsity in them
see here for the research we are expanding up

Much appreciated ! Is there a plan to make your research open source ? I don't expect a good accuracy for post-training sparsity 2:4. Have you guys looked into other unstructured pruning methods or dynamic pruning methods?

Yes: https://huggingface.co/collections/neuralmagic/sparse-foundational-llama-2-models-65f48cec6396309f02e74d21

• 2:4 models coming soon

We are currently focused on pruning Llama-3 on our cluster

We have done extensive research into pruning with our friends at IST.

@AniZpZ are you on discord?

Yes, how can I reach you guys on discord?

[robertgshaw2-neuralmagic]

@AniZpZ here's current hacked up script we have
Note: accuracy will not be great for post-training sparsity 2:4.
We have been investing in making sparse foundational models that have sparsity in them
see here for the research we are expanding up

Much appreciated ! Is there a plan to make your research open source ? I don't expect a good accuracy for post-training sparsity 2:4. Have you guys looked into other unstructured pruning methods or dynamic pruning methods?

Yes: https://huggingface.co/collections/neuralmagic/sparse-foundational-llama-2-models-65f48cec6396309f02e74d21

• 2:4 models coming soon

We are currently focused on pruning Llama-3 on our cluster
We have done extensive research into pruning with our friends at IST.
@AniZpZ are you on discord?

Yes, how can I reach you guys on discord?

Im on the vllm-server (https://discord.gg/3eCXvqVu) - robertgshaw-neural-magic

Send me a note