You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Just a curious question I suppose!
GPTQ 4bit - https://github.com/qwopqwop200/GPTQ-for-LLaMa
Suppose someone eventually finetunes 175B OPT model, with loras or regular finetunng. or perhaps the BLOOM or BLOOMZ model, would running inference be possible with GPTQ to allow the model to be run on 4gbvram and 50gb dram?
The text was updated successfully, but these errors were encountered:
The compression in FlexGen has computation overhead, so it is not always better to turn it on. For large models like 175B which involves disk swap, it is usually better to turn on both weights and cache compression.
GPTQ 4bit has not been implemented in FlexGen.
Even you use 4bit, the weights of an 175B model need to occupy ~90G memory. 4GB vram and 50GB dram is not sufficient.
Just a curious question I suppose!
GPTQ 4bit - https://github.com/qwopqwop200/GPTQ-for-LLaMa
Suppose someone eventually finetunes 175B OPT model, with loras or regular finetunng. or perhaps the BLOOM or BLOOMZ model, would running inference be possible with GPTQ to allow the model to be run on 4gbvram and 50gb dram?
The text was updated successfully, but these errors were encountered: