Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix] quantization properties for lmi dist and hf acc #1318

Merged
merged 1 commit into from
Nov 15, 2023

Conversation

sindhuvahinis
Copy link
Contributor

Description

Found a bug while testing quantization with new properties for HF Accelerate and lmi-dist

LMI Dist rolling batch takes in the parsed properties instead of pydantic, so new bitsandbytes8 needs to be handled in rolling batch handler.

Testing:
Tested HF and LMI Dist in my machine with bitsandbytes option
Tested TnX quantization in my machine , works

vllm, deepseed, trtllm has no changes for quantization

@sindhuvahinis sindhuvahinis self-assigned this Nov 15, 2023
@sindhuvahinis sindhuvahinis removed their assignment Nov 15, 2023
@sindhuvahinis sindhuvahinis merged commit f0ea80b into deepjavalibrary:master Nov 15, 2023
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants