New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[community] add more data types support to ipex-llm
llm integration
#12635
Conversation
add more data types support
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good to me -- can you bump the package version of the integration to v0.1.1. so that it gets published?
bumped version and add some more info in README. |
bump version and update README
IpexLLM
will load the model in int4 format. This PR adds more data types support such assym_in5
,sym_int8
, etc. Data formats like NF3, NF4, FP4 and FP8 are only supported on GPU and will be added in future PR.