Skip to content

Adding support for GPTQ models #103

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 13 commits into from
Sep 13, 2024
Merged

Adding support for GPTQ models #103

merged 13 commits into from
Sep 13, 2024

Conversation

@quic-amitraj
Copy link
Contributor

@quic-amitraj quic-amitraj commented Sep 7, 2024 •
edited
Loading

  1. Adding support for GPTQ quantized models.
  2. Shifted all the common code to qunatizer_utils.py.
  3. Added test for GPTQ model.

@quic-amitraj quic-amitraj changed the title Adding support for gptq models Adding support for GPTQ models Sep 7, 2024
@quic-amitraj quic-amitraj marked this pull request as draft September 7, 2024 19:43
@quic-amitraj quic-amitraj marked this pull request as ready for review September 8, 2024 12:45
ochougul
ochougul requested changes Sep 10, 2024
View reviewed changes
@quic-amitraj
Adding support for gptq models
c4d3d24 
Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>
@quic-amitraj
Code cleaning and formating
d5b1edd 
Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>
@quic-amitraj
ruff format and fixed some bug
4b3c190 
Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>
@quic-amitraj
Added tests for gptq
aa01d08 
Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>
@quic-amitraj
Bug-fix-1
fec210d 
Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>
@quic-amitraj
fixed bugs-2
06764f3 
Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>
@quic-amitraj
fixed bug-3
541a3ef 
Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>
@quic-amitraj
Added docstring
e8993a1 
Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>
@quic-amitraj
Addressed comments
0388f80 
Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>
@quic-amitraj
Addressed comments
ba9088c 
Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>
@quic-amitraj
fixed bugs-3
695bc0e 
Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>
@quic-amitraj
ruff check and format
e49d815 
Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>
@quic-amitraj
Addressed comments-3
64663ba 
Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>
@quic-amitraj
Copy link
Contributor Author

quic-amitraj commented Sep 13, 2024 •
edited
Loading

Output for model -https://huggingface.co/TheBloke/Llama-2-7B-Chat-GPTQ

  1. Without TS-
    image
  2. TS with 4 devices-
    image
  3. With CB with batch size=3-
    image

@ochougul ochougul merged commit 14b4de9 into quic:awq+gptq Sep 13, 2024
ochougul pushed a commit that referenced this pull request Sep 13, 2024
@quic-amitraj @ochougul
Adding support for GPTQ models (#103)
6dba3db 
* Adding support for gptq models

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Code cleaning and formating

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* ruff format and fixed some bug

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Added tests for gptq

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Bug-fix-1

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* fixed bugs-2

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* fixed bug-3

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Added docstring

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Addressed comments

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Addressed comments

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* fixed bugs-3

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* ruff check and format

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Addressed comments-3

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

---------

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>
Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>
ochougul pushed a commit that referenced this pull request Sep 13, 2024
@quic-amitraj @ochougul
Adding support for GPTQ models (#103)
6e9f158 
* Adding support for gptq models

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Code cleaning and formating

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* ruff format and fixed some bug

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Added tests for gptq

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Bug-fix-1

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* fixed bugs-2

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* fixed bug-3

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Added docstring

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Addressed comments

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Addressed comments

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* fixed bugs-3

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* ruff check and format

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Addressed comments-3

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

---------

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>
Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>
ochougul added a commit that referenced this pull request Sep 13, 2024
@ochougul @quic-amitraj
AWQ+GPTQ (#101)
afb4645 
* Awq feature (#100)

* added preprocess layer before loading quantized awq weights

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* added onnx export

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* added ScaledActivation class

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* refactoring the code to right places and added one single test for now

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* cleaned code

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* added proper tests, added decorator for updating quantizers, cleaned code

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* fixed CLI

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* added auto file for decorator

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

---------

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* bugfix for tests

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* fixed tests for AWQ model

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* Adding support for GPTQ models (#103)

* Adding support for gptq models

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Code cleaning and formating

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* ruff format and fixed some bug

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Added tests for gptq

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Bug-fix-1

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* fixed bugs-2

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* fixed bug-3

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Added docstring

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Addressed comments

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Addressed comments

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* fixed bugs-3

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* ruff check and format

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Addressed comments-3

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

---------

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>
Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* added liscence at top for missing file

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* added export_and_compile and fixed bugs

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* removed GPTQ test

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* removed threading from pytest

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

---------

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>
Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>
Co-authored-by: Amit Raj <168538872+quic-amitraj@users.noreply.github.com>
quic-amitraj added a commit to quic-amitraj/efficient-transformers that referenced this pull request Sep 16, 2024
@ochougul @quic-amitraj
AWQ+GPTQ (quic#101)
f2b09d2 
* Awq feature (quic#100)

* added preprocess layer before loading quantized awq weights

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* added onnx export

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* added ScaledActivation class

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* refactoring the code to right places and added one single test for now

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* cleaned code

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* added proper tests, added decorator for updating quantizers, cleaned code

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* fixed CLI

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* added auto file for decorator

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

---------

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* bugfix for tests

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* fixed tests for AWQ model

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* Adding support for GPTQ models (quic#103)

* Adding support for gptq models

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Code cleaning and formating

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* ruff format and fixed some bug

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Added tests for gptq

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Bug-fix-1

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* fixed bugs-2

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* fixed bug-3

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Added docstring

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Addressed comments

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Addressed comments

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* fixed bugs-3

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* ruff check and format

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Addressed comments-3

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

---------

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>
Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* added liscence at top for missing file

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* added export_and_compile and fixed bugs

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* removed GPTQ test

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* removed threading from pytest

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

---------

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>
Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>
Co-authored-by: Amit Raj <168538872+quic-amitraj@users.noreply.github.com>
Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>
quic-amitraj added a commit to quic-amitraj/efficient-transformers that referenced this pull request Sep 16, 2024
@ochougul @quic-amitraj
AWQ+GPTQ (quic#101)
b745d2d 
* Awq feature (quic#100)

* added preprocess layer before loading quantized awq weights

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* added onnx export

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* added ScaledActivation class

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* refactoring the code to right places and added one single test for now

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* cleaned code

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* added proper tests, added decorator for updating quantizers, cleaned code

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* fixed CLI

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* added auto file for decorator

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

---------

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* bugfix for tests

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* fixed tests for AWQ model

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* Adding support for GPTQ models (quic#103)

* Adding support for gptq models

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Code cleaning and formating

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* ruff format and fixed some bug

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Added tests for gptq

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Bug-fix-1

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* fixed bugs-2

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* fixed bug-3

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Added docstring

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Addressed comments

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Addressed comments

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* fixed bugs-3

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* ruff check and format

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

* Addressed comments-3

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>

---------

Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>
Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* added liscence at top for missing file

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* added export_and_compile and fixed bugs

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* removed GPTQ test

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

* removed threading from pytest

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>

---------

Signed-off-by: Onkar Chougule <quic_ochougul@quicinc.com>
Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>
Co-authored-by: Amit Raj <168538872+quic-amitraj@users.noreply.github.com>
Signed-off-by: Amit Raj <quic_amitraj@quicinc.com>
@quic-amitraj quic-amitraj deleted the gptq_support branch November 21, 2024 04:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ochougul ochougul ochougul approved these changes

@irajagop irajagop Awaiting requested review from irajagop

@quic-akuruvil quic-akuruvil Awaiting requested review from quic-akuruvil

@anujgupt-github anujgupt-github Awaiting requested review from anujgupt-github

@vbaddi vbaddi Awaiting requested review from vbaddi

@quic-mamta quic-mamta Awaiting requested review from quic-mamta

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@quic-amitraj @ochougul