Improved performance of decoders #354
Conversation
|
The documentation is not available anymore as the PR was closed or merged. |
| for i, data in enumerate(calibration_dataloader): | ||
| self.model.generate(**data, max_new_tokens=10) | ||
| for _, data in enumerate(calibration_dataloader): | ||
| self.model.generate(**data, max_new_tokens=100) |
There was a problem hiding this comment.
Is this modification added to reduce accuracy degradation resulting from quantization? If yes, what did you observe when varying this parameter ?
There was a problem hiding this comment.
In the process, and will update a bit later.
| def __getattr__(self, attr): | ||
| if attr in self.__dict__: | ||
| return getattr(self, attr) | ||
| return getattr(self.request, attr) | ||
|
|
||
| self.model.request = InferRequestWrapper(self.model.request) | ||
| for i, data in enumerate(calibration_dataloader): | ||
| self.model.generate(**data, max_new_tokens=10) | ||
| for _, data in enumerate(calibration_dataloader): |
There was a problem hiding this comment.
Not related to the PR, but what do you think about uniformizing how quantization is applied on causal langage models depending on whether the user gives a torch.nn.Module or a OVBaseDecoderModel (the number of generation steps is currently not the same). We could also instantiate an OVModel in the from_pretrained method when the given model is a PreTrainedModel
There was a problem hiding this comment.
It is hard to accomplish this with the current NNCF PTQ API implementation we have for PyTorch. I think we should deprecate PTQ for PyTorch at some point because it also introduces ambiguity for the user about what workflow to use for quantization.
Co-authored-by: Ella Charlaix <80481427+echarlaix@users.noreply.github.com>
|
I think it is ready for merge. |
* Improved performance of decoders * Improved performance of Seq2seq models * Style * Adjusted quantization logic * Style * Temporal changes * Temporal * Make it working * Some improvements * Style * Update optimum/intel/openvino/modeling_decoder.py Co-authored-by: Ella Charlaix <80481427+echarlaix@users.noreply.github.com> --------- Co-authored-by: Ella Charlaix <80481427+echarlaix@users.noreply.github.com>
* Improved performance of decoders * Improved performance of Seq2seq models * Style * Adjusted quantization logic * Style * Temporal changes * Temporal * Make it working * Some improvements * Style * Update optimum/intel/openvino/modeling_decoder.py Co-authored-by: Ella Charlaix <80481427+echarlaix@users.noreply.github.com> --------- Co-authored-by: Ella Charlaix <80481427+echarlaix@users.noreply.github.com>
Improved performance of decoders: GPT-like, bloom, etc., and Seq2seq models.
Observed a significant speedup of long sequences, e.g. Dolly-3B, 500 tokens: +30% when running on a client CPU.