-
Notifications
You must be signed in to change notification settings - Fork 106
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improved performance of decoders #354
Conversation
The documentation is not available anymore as the PR was closed or merged. |
for i, data in enumerate(calibration_dataloader): | ||
self.model.generate(**data, max_new_tokens=10) | ||
for _, data in enumerate(calibration_dataloader): | ||
self.model.generate(**data, max_new_tokens=100) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this modification added to reduce accuracy degradation resulting from quantization? If yes, what did you observe when varying this parameter ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the process, and will update a bit later.
def __getattr__(self, attr): | ||
if attr in self.__dict__: | ||
return getattr(self, attr) | ||
return getattr(self.request, attr) | ||
|
||
self.model.request = InferRequestWrapper(self.model.request) | ||
for i, data in enumerate(calibration_dataloader): | ||
self.model.generate(**data, max_new_tokens=10) | ||
for _, data in enumerate(calibration_dataloader): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not related to the PR, but what do you think about uniformizing how quantization is applied on causal langage models depending on whether the user gives a torch.nn.Module
or a OVBaseDecoderModel
(the number of generation steps is currently not the same). We could also instantiate an OVModel
in the from_pretrained
method when the given model is a PreTrainedModel
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is hard to accomplish this with the current NNCF PTQ API implementation we have for PyTorch. I think we should deprecate PTQ for PyTorch at some point because it also introduces ambiguity for the user about what workflow to use for quantization.
Co-authored-by: Ella Charlaix <80481427+echarlaix@users.noreply.github.com>
I think it is ready for merge. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🚀 🔥
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks a lot @AlexKoff88
* Improved performance of decoders * Improved performance of Seq2seq models * Style * Adjusted quantization logic * Style * Temporal changes * Temporal * Make it working * Some improvements * Style * Update optimum/intel/openvino/modeling_decoder.py Co-authored-by: Ella Charlaix <80481427+echarlaix@users.noreply.github.com> --------- Co-authored-by: Ella Charlaix <80481427+echarlaix@users.noreply.github.com>
* Improved performance of decoders * Improved performance of Seq2seq models * Style * Adjusted quantization logic * Style * Temporal changes * Temporal * Make it working * Some improvements * Style * Update optimum/intel/openvino/modeling_decoder.py Co-authored-by: Ella Charlaix <80481427+echarlaix@users.noreply.github.com> --------- Co-authored-by: Ella Charlaix <80481427+echarlaix@users.noreply.github.com>
Improved performance of decoders: GPT-like, bloom, etc., and Seq2seq models.
Observed a significant speedup of long sequences, e.g. Dolly-3B, 500 tokens: +30% when running on a client CPU.