Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add phi3 support #183

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Add phi3 support #183

wants to merge 2 commits into from
+131 −5

Conversation

pprp
Copy link

@pprp pprp commented May 8, 2024

This pull request introduces support for the phi3 architecture within the llm-awq.

The transformer package for phi3 is not released. So I developed it using 4.41.0.dev0.

This PR works fine on RTX4090 and Jetson Orin 8G.

@casper-hansen casper-hansen mentioned this pull request May 8, 2024
Open
@Abhis-123
Copy link

Abhis-123 commented May 11, 2024 •
edited

HI @pprp Thanks for putting effort, can you share more about the requirements to build this pr, I want to use this branch for quantization..
Because in your pr it's still transformers==4.36.2

@pprp
Copy link
Author

pprp commented May 12, 2024

@Abhis-123 Here is the environment in jetson Orin:

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main
_openmp_mutex             5.1                      51_gnu
absl-py                   2.1.0                    pypi_0    pypi
accelerate                0.28.0                   pypi_0    pypi
aiofiles                  23.2.1                   pypi_0    pypi
aiohttp                   3.9.3                    pypi_0    pypi
aiosignal                 1.3.1                    pypi_0    pypi
altair                    5.2.0                    pypi_0    pypi
anyio                     4.3.0                    pypi_0    pypi
async-timeout             4.0.3                    pypi_0    pypi
attributedict             0.3.0                    pypi_0    pypi
attrs                     23.2.0                   pypi_0    pypi
awq                       0.1.0                    pypi_0    pypi
awq-inference-engine      0.0.0                    pypi_0    pypi
blessings                 1.7                      pypi_0    pypi
bzip2                     1.0.8                h998d150_5
ca-certificates           2024.3.11            hd43f75c_0
cachetools                5.3.3                    pypi_0    pypi
certifi                   2024.2.2                 pypi_0    pypi
chardet                   5.2.0                    pypi_0    pypi
charset-normalizer        3.3.2                    pypi_0    pypi
click                     8.1.7                    pypi_0    pypi
codecov                   2.1.13                   pypi_0    pypi
colorama                  0.4.6                    pypi_0    pypi
coloredlogs               15.0.1                   pypi_0    pypi
colour-runner             0.1.1                    pypi_0    pypi
contourpy                 1.1.1                    pypi_0    pypi
coverage                  7.4.4                    pypi_0    pypi
cycler                    0.12.1                   pypi_0    pypi
dataproperty              1.0.1                    pypi_0    pypi
datasets                  2.18.0                   pypi_0    pypi
deepdiff                  6.7.1                    pypi_0    pypi
dill                      0.3.8                    pypi_0    pypi
distlib                   0.3.8                    pypi_0    pypi
distro                    1.9.0                    pypi_0    pypi
dockerhub-api             0.5                      pypi_0    pypi
exceptiongroup            1.2.0                    pypi_0    pypi
fastapi                   0.110.0                  pypi_0    pypi
ffmpy                     0.3.2                    pypi_0    pypi
filelock                  3.13.1                   pypi_0    pypi
fonttools                 4.50.0                   pypi_0    pypi
frozenlist                1.4.1                    pypi_0    pypi
fsspec                    2024.2.0                 pypi_0    pypi
furl                      2.1.3                    pypi_0    pypi
gguf                      0.6.0                    pypi_0    pypi
gradio                    3.35.2                   pypi_0    pypi
gradio-client             0.2.9                    pypi_0    pypi
h11                       0.14.0                   pypi_0    pypi
hf-transfer               0.1.6                    pypi_0    pypi
httpcore                  1.0.4                    pypi_0    pypi
httpx                     0.27.0                   pypi_0    pypi
huggingface-hub           0.23.0                   pypi_0    pypi
humanfriendly             10.0                     pypi_0    pypi
idna                      3.6                      pypi_0    pypi
importlib-resources       5.13.0                   pypi_0    pypi
inspecta                  0.1.3                    pypi_0    pypi
jinja2                    3.1.3                    pypi_0    pypi
joblib                    1.3.2                    pypi_0    pypi
jsonlines                 4.0.0                    pypi_0    pypi
jsonschema                4.21.1                   pypi_0    pypi
jsonschema-specifications 2023.12.1                pypi_0    pypi
kiwisolver                1.4.5                    pypi_0    pypi
ld_impl_linux-aarch64     2.38                 h8131f2d_1
libffi                    3.4.4                h419075a_0
libgcc-ng                 11.2.0               h1234567_1
libgomp                   11.2.0               h1234567_1
libstdcxx-ng              11.2.0               h1234567_1
libuuid                   1.41.5               h998d150_0
linkify-it-py             2.0.3                    pypi_0    pypi
lm-eval                   0.3.0                    pypi_0    pypi
markdown-it-py            2.2.0                    pypi_0    pypi
markupsafe                2.1.5                    pypi_0    pypi
matplotlib                3.7.5                    pypi_0    pypi
mbstrdecoder              1.1.3                    pypi_0    pypi
mdit-py-plugins           0.3.3                    pypi_0    pypi
mdurl                     0.1.2                    pypi_0    pypi
mpmath                    1.3.0                    pypi_0    pypi
multidict                 6.0.5                    pypi_0    pypi
multiprocess              0.70.16                  pypi_0    pypi
ncurses                   6.4                  h419075a_0
networkx                  3.1                      pypi_0    pypi
nltk                      3.8.1                    pypi_0    pypi
numexpr                   2.8.6                    pypi_0    pypi
numpy                     1.24.4                   pypi_0    pypi
openai                    1.14.2                   pypi_0    pypi
openssl                   3.0.13               h2f4d8fa_0
ordered-set               4.1.0                    pypi_0    pypi
orderedmultidict          1.0.1                    pypi_0    pypi
orjson                    3.9.15                   pypi_0    pypi
packaging                 24.0                     pypi_0    pypi
pandas                    2.0.3                    pypi_0    pypi
pathvalidate              3.2.0                    pypi_0    pypi
pillow                    10.2.0                   pypi_0    pypi
pip                       24.0                     pypi_0    pypi
pkgutil-resolve-name      1.3.10                   pypi_0    pypi
platformdirs              4.2.0                    pypi_0    pypi
pluggy                    1.4.0                    pypi_0    pypi
portalocker               2.8.2                    pypi_0    pypi
protobuf                  5.26.0                   pypi_0    pypi
psutil                    5.9.8                    pypi_0    pypi
pyarrow                   15.0.2                   pypi_0    pypi
pyarrow-hotfix            0.6                      pypi_0    pypi
pybind11                  2.11.1                   pypi_0    pypi
pycountry                 23.12.11                 pypi_0    pypi
pydantic                  1.10.14                  pypi_0    pypi
pydub                     0.25.1                   pypi_0    pypi
pygments                  2.17.2                   pypi_0    pypi
pyparsing                 3.1.2                    pypi_0    pypi
pyproject-api             1.6.1                    pypi_0    pypi
pytablewriter             1.2.0                    pypi_0    pypi
python                    3.8.19               h4bb2201_0
python-dateutil           2.9.0.post0              pypi_0    pypi
python-multipart          0.0.9                    pypi_0    pypi
pytz                      2024.1                   pypi_0    pypi
pyyaml                    6.0.1                    pypi_0    pypi
readline                  8.2                  h998d150_0
referencing               0.34.0                   pypi_0    pypi
regex                     2023.12.25               pypi_0    pypi
requests                  2.31.0                   pypi_0    pypi
rootpath                  0.1.1                    pypi_0    pypi
rouge-score               0.1.2                    pypi_0    pypi
rpds-py                   0.18.0                   pypi_0    pypi
sacrebleu                 1.5.0                    pypi_0    pypi
safetensors               0.4.2                    pypi_0    pypi
scikit-learn              1.3.2                    pypi_0    pypi
scipy                     1.10.1                   pypi_0    pypi
semantic-version          2.10.0                   pypi_0    pypi
sentencepiece             0.2.0                    pypi_0    pypi
setuptools                68.2.2           py38hd43f75c_0
six                       1.16.0                   pypi_0    pypi
sniffio                   1.3.1                    pypi_0    pypi
sqlite                    3.41.2               h998d150_0
sqlitedict                2.1.0                    pypi_0    pypi
starlette                 0.36.3                   pypi_0    pypi
sympy                     1.12                     pypi_0    pypi
tabledata                 1.3.3                    pypi_0    pypi
tcolorpy                  0.1.4                    pypi_0    pypi
termcolor                 2.4.0                    pypi_0    pypi
texttable                 1.7.0                    pypi_0    pypi
threadpoolctl             3.4.0                    pypi_0    pypi
tk                        8.6.12               h241ca14_0
tokenizers                0.19.1                   pypi_0    pypi
toml                      0.10.2                   pypi_0    pypi
tomli                     2.0.1                    pypi_0    pypi
toolz                     0.12.1                   pypi_0    pypi
torch                     2.0.0+nv23.5             pypi_0    pypi
tox                       4.14.1                   pypi_0    pypi
tqdm                      4.66.2                   pypi_0    pypi
tqdm-multiprocess         0.0.11                   pypi_0    pypi
transformers              4.41.0.dev0              pypi_0    pypi
typepy                    1.3.2                    pypi_0    pypi
typing-extensions         4.10.0                   pypi_0    pypi
tzdata                    2024.1                   pypi_0    pypi
uc-micro-py               1.0.3                    pypi_0    pypi
urllib3                   2.2.1                    pypi_0    pypi
uvicorn                   0.29.0                   pypi_0    pypi
virtualenv                20.25.1                  pypi_0    pypi
websockets                12.0                     pypi_0    pypi
wget                      3.2                      pypi_0    pypi
wheel                     0.41.2           py38hd43f75c_0
xxhash                    3.4.1                    pypi_0    pypi
xz                        5.4.6                h998d150_0
yarl                      1.9.4                    pypi_0    pypi
zipp                      3.18.1                   pypi_0    pypi
zlib                      1.2.13               h998d150_0
zstandard                 0.22.0                   pypi_0    pypi
(awq) iota@iota-desktop:~$ conda list > env.txt
(awq) iota@iota-desktop:~$ cat env.txt
# packages in environment at /home/iota/anaconda3/envs/awq:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main
_openmp_mutex             5.1                      51_gnu
absl-py                   2.1.0                    pypi_0    pypi
accelerate                0.28.0                   pypi_0    pypi
aiofiles                  23.2.1                   pypi_0    pypi
aiohttp                   3.9.3                    pypi_0    pypi
aiosignal                 1.3.1                    pypi_0    pypi
altair                    5.2.0                    pypi_0    pypi
anyio                     4.3.0                    pypi_0    pypi
async-timeout             4.0.3                    pypi_0    pypi
attributedict             0.3.0                    pypi_0    pypi
attrs                     23.2.0                   pypi_0    pypi
awq                       0.1.0                    pypi_0    pypi
awq-inference-engine      0.0.0                    pypi_0    pypi
blessings                 1.7                      pypi_0    pypi
bzip2                     1.0.8                h998d150_5
ca-certificates           2024.3.11            hd43f75c_0
cachetools                5.3.3                    pypi_0    pypi
certifi                   2024.2.2                 pypi_0    pypi
chardet                   5.2.0                    pypi_0    pypi
charset-normalizer        3.3.2                    pypi_0    pypi
click                     8.1.7                    pypi_0    pypi
codecov                   2.1.13                   pypi_0    pypi
colorama                  0.4.6                    pypi_0    pypi
coloredlogs               15.0.1                   pypi_0    pypi
colour-runner             0.1.1                    pypi_0    pypi
contourpy                 1.1.1                    pypi_0    pypi
coverage                  7.4.4                    pypi_0    pypi
cycler                    0.12.1                   pypi_0    pypi
dataproperty              1.0.1                    pypi_0    pypi
datasets                  2.18.0                   pypi_0    pypi
deepdiff                  6.7.1                    pypi_0    pypi
dill                      0.3.8                    pypi_0    pypi
distlib                   0.3.8                    pypi_0    pypi
distro                    1.9.0                    pypi_0    pypi
dockerhub-api             0.5                      pypi_0    pypi
exceptiongroup            1.2.0                    pypi_0    pypi
fastapi                   0.110.0                  pypi_0    pypi
ffmpy                     0.3.2                    pypi_0    pypi
filelock                  3.13.1                   pypi_0    pypi
fonttools                 4.50.0                   pypi_0    pypi
frozenlist                1.4.1                    pypi_0    pypi
fsspec                    2024.2.0                 pypi_0    pypi
furl                      2.1.3                    pypi_0    pypi
gguf                      0.6.0                    pypi_0    pypi
gradio                    3.35.2                   pypi_0    pypi
gradio-client             0.2.9                    pypi_0    pypi
h11                       0.14.0                   pypi_0    pypi
hf-transfer               0.1.6                    pypi_0    pypi
httpcore                  1.0.4                    pypi_0    pypi
httpx                     0.27.0                   pypi_0    pypi
huggingface-hub           0.23.0                   pypi_0    pypi
humanfriendly             10.0                     pypi_0    pypi
idna                      3.6                      pypi_0    pypi
importlib-resources       5.13.0                   pypi_0    pypi
inspecta                  0.1.3                    pypi_0    pypi
jinja2                    3.1.3                    pypi_0    pypi
joblib                    1.3.2                    pypi_0    pypi
jsonlines                 4.0.0                    pypi_0    pypi
jsonschema                4.21.1                   pypi_0    pypi
jsonschema-specifications 2023.12.1                pypi_0    pypi
kiwisolver                1.4.5                    pypi_0    pypi
ld_impl_linux-aarch64     2.38                 h8131f2d_1
libffi                    3.4.4                h419075a_0
libgcc-ng                 11.2.0               h1234567_1
libgomp                   11.2.0               h1234567_1
libstdcxx-ng              11.2.0               h1234567_1
libuuid                   1.41.5               h998d150_0
linkify-it-py             2.0.3                    pypi_0    pypi
lm-eval                   0.3.0                    pypi_0    pypi
markdown-it-py            2.2.0                    pypi_0    pypi
markupsafe                2.1.5                    pypi_0    pypi
matplotlib                3.7.5                    pypi_0    pypi
mbstrdecoder              1.1.3                    pypi_0    pypi
mdit-py-plugins           0.3.3                    pypi_0    pypi
mdurl                     0.1.2                    pypi_0    pypi
mpmath                    1.3.0                    pypi_0    pypi
multidict                 6.0.5                    pypi_0    pypi
multiprocess              0.70.16                  pypi_0    pypi
ncurses                   6.4                  h419075a_0
networkx                  3.1                      pypi_0    pypi
nltk                      3.8.1                    pypi_0    pypi
numexpr                   2.8.6                    pypi_0    pypi
numpy                     1.24.4                   pypi_0    pypi
openai                    1.14.2                   pypi_0    pypi
openssl                   3.0.13               h2f4d8fa_0
ordered-set               4.1.0                    pypi_0    pypi
orderedmultidict          1.0.1                    pypi_0    pypi
orjson                    3.9.15                   pypi_0    pypi
packaging                 24.0                     pypi_0    pypi
pandas                    2.0.3                    pypi_0    pypi
pathvalidate              3.2.0                    pypi_0    pypi
pillow                    10.2.0                   pypi_0    pypi
pip                       24.0                     pypi_0    pypi
pkgutil-resolve-name      1.3.10                   pypi_0    pypi
platformdirs              4.2.0                    pypi_0    pypi
pluggy                    1.4.0                    pypi_0    pypi
portalocker               2.8.2                    pypi_0    pypi
protobuf                  5.26.0                   pypi_0    pypi
psutil                    5.9.8                    pypi_0    pypi
pyarrow                   15.0.2                   pypi_0    pypi
pyarrow-hotfix            0.6                      pypi_0    pypi
pybind11                  2.11.1                   pypi_0    pypi
pycountry                 23.12.11                 pypi_0    pypi
pydantic                  1.10.14                  pypi_0    pypi
pydub                     0.25.1                   pypi_0    pypi
pygments                  2.17.2                   pypi_0    pypi
pyparsing                 3.1.2                    pypi_0    pypi
pyproject-api             1.6.1                    pypi_0    pypi
pytablewriter             1.2.0                    pypi_0    pypi
python                    3.8.19               h4bb2201_0
python-dateutil           2.9.0.post0              pypi_0    pypi
python-multipart          0.0.9                    pypi_0    pypi
pytz                      2024.1                   pypi_0    pypi
pyyaml                    6.0.1                    pypi_0    pypi
readline                  8.2                  h998d150_0
referencing               0.34.0                   pypi_0    pypi
regex                     2023.12.25               pypi_0    pypi
requests                  2.31.0                   pypi_0    pypi
rootpath                  0.1.1                    pypi_0    pypi
rouge-score               0.1.2                    pypi_0    pypi
rpds-py                   0.18.0                   pypi_0    pypi
sacrebleu                 1.5.0                    pypi_0    pypi
safetensors               0.4.2                    pypi_0    pypi
scikit-learn              1.3.2                    pypi_0    pypi
scipy                     1.10.1                   pypi_0    pypi
semantic-version          2.10.0                   pypi_0    pypi
sentencepiece             0.2.0                    pypi_0    pypi
setuptools                68.2.2           py38hd43f75c_0
six                       1.16.0                   pypi_0    pypi
sniffio                   1.3.1                    pypi_0    pypi
sqlite                    3.41.2               h998d150_0
sqlitedict                2.1.0                    pypi_0    pypi
starlette                 0.36.3                   pypi_0    pypi
sympy                     1.12                     pypi_0    pypi
tabledata                 1.3.3                    pypi_0    pypi
tcolorpy                  0.1.4                    pypi_0    pypi
termcolor                 2.4.0                    pypi_0    pypi
texttable                 1.7.0                    pypi_0    pypi
threadpoolctl             3.4.0                    pypi_0    pypi
tk                        8.6.12               h241ca14_0
tokenizers                0.19.1                   pypi_0    pypi
toml                      0.10.2                   pypi_0    pypi
tomli                     2.0.1                    pypi_0    pypi
toolz                     0.12.1                   pypi_0    pypi
torch                     2.0.0+nv23.5             pypi_0    pypi
tox                       4.14.1                   pypi_0    pypi
tqdm                      4.66.2                   pypi_0    pypi
tqdm-multiprocess         0.0.11                   pypi_0    pypi
transformers              4.41.0.dev0              pypi_0    pypi
typepy                    1.3.2                    pypi_0    pypi
typing-extensions         4.10.0                   pypi_0    pypi
tzdata                    2024.1                   pypi_0    pypi
uc-micro-py               1.0.3                    pypi_0    pypi
urllib3                   2.2.1                    pypi_0    pypi
uvicorn                   0.29.0                   pypi_0    pypi
virtualenv                20.25.1                  pypi_0    pypi
websockets                12.0                     pypi_0    pypi
wget                      3.2                      pypi_0    pypi
wheel                     0.41.2           py38hd43f75c_0
xxhash                    3.4.1                    pypi_0    pypi
xz                        5.4.6                h998d150_0
yarl                      1.9.4                    pypi_0    pypi
zipp                      3.18.1                   pypi_0    pypi
zlib                      1.2.13               h998d150_0
zstandard                 0.22.0                   pypi_0    pypi

@Abhis-123
Copy link

@pprp experimented with it , the quantized phi3 comes in just 54mb. Is there something i missed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@pprp @Abhis-123