Update huggingface.py

[Archit-Kohli]

Added quantization support using bitsandbytes in application/llms/huggingface.py using simple q parameter for quantization support as in #409

• What kind of change does this PR introduce? Feature Addition

• Why was this change needed? Since the model is too large, quantization support is needed

• Other information: Added bitsandbytes support for loading quantized model

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
docs-gpt	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 5, 2023 8:58am
nextra-docsgpt	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 5, 2023 8:58am

[codecov]

Codecov Report

Merging #425 (7ce1dc9) into main (4be0c1c) will decrease coverage by 0.04%.
Report is 14 commits behind head on main.
The diff coverage is 10.00%.

@@            Coverage Diff             @@
##             main     #425      +/-   ##
==========================================
- Coverage   16.01%   15.97%   -0.04%     
==========================================
  Files          59       59              
  Lines        2667     2673       +6     
==========================================
  Hits          427      427              
- Misses       2240     2246       +6     

Files	Coverage Δ
application/llm/huggingface.py	21.73% <10.00%> (-7.68%)	⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[dartpain]

Perfect, thank you! Just make sure you import torch in the if q: section please

[Archit-Kohli]

I have done that just now, please check

[dartpain]

@holopin-bot @Archit-Kohli

[holopin-bot]

Congratulations @Archit-Kohli, you just earned a badge! Here it is: https://holopin.io/claim/clnd0foph78500flavc01bcao

This badge can only be claimed by you, so make sure that your GitHub account is linked to your Holopin account. You can manage those preferences here: https://holopin.io/account.
Or if you're new to Holopin, you can simply sign up with GitHub, which will do the trick!