-
Notifications
You must be signed in to change notification settings - Fork 26.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Translation Model in ONNX: Choosable Output Formats #9784
Comments
Hello, Thanks for you work on that @oborchers ! I also saw the notebook on SentenceTransformers and it helped a lot ! Any status about this feature ? I also need to run models in onnx but most of them need to call a Thanks a lot in advance, |
Hi @lerezell! Welcome! Glad the notebook helped. Not from my side, unfortunately. I had postponed the issue from our side, because we had more pressing stuff to work on. But lately, the need for this feature starts to become larger as well at our company. Does your solution of porting the Edit: In an initial implementation, it should be perfectly sufficient to port just the Downsides of this are, that, there needs to be some way of defining the arguments of Best regards and have a nice eastern |
Hi @oborchers, I still haven't implemented "my" solution as I wanted to know if there was any other solution than writing all the logic again. For the I need to focus on other things right now but I definitely keep an eye open for that ! Have a great day |
Hi, any update on how to export full pipelines to onnx? For now, we're still obliged to keep a custom/hugging face lib code to handle the "post output embeddings" logic.... Thanks in advance, |
Hi @ierezell! Sorry for not coming back on the issue. To be honest, for our use case there are quite a few problems we've encountered in exporting full pipelines to ONNX:
This blog post from Microsoft may help though: |
Hi @oborchers, thanks a lot for the feedback! Onnx is nice to be able to change stack for me (javascript etc...) but in the light of what you're saying it will be better to keep my GPU inference server. Thanks a lot, |
Hi, Is there an alternative to onnx that you'd recommend? The able to keep and manipulate past_key_values is the most crucial part that I cannot find for many inference optimizations. Thank you! |
🚀 Feature request
I am requesting to provide an option to specify the output format for the
translation_xx_to_yy
export to ONNX models. Currently, the output of convert_graph_to_onnx.convert will provide the raw tensors as output (working prototype code under #9722)Motivation
When putting the models into production it would be great if one could chose, whether one wants to have the actual tensors or the output-tokens returned when exporting a translation pipeline to ONNX. Thereby, one is not forced to do a custom re-implementation of the model.generate function, which then uses the ONNX model instead of the torch one.
As for now, the part which is could be replaced by an ONNX inference session lives under the model.generate function. Using this in production would mean to keep a TranslationPipeline object with all corresponding model information and config plus an ONNX inference session.
Your contribution
There may be multiple solutions to this problem:
User-specific re-implementation of model.generate (This is what Ill try to accomplish in the future)
Is it possible to rewrite the code under model.generate to full torch? Then it should be possible to create a custom model for all translation models, that just places this "generate layer" on top of it. I have provided an example here which adds a simple pooling layer on an already extant transformers model. (That would require more study from my side to develop a prototype and follows step 1)
Provide support for the ort-customops library by Microsoft. Essentially, this enables ONNX to handle strings (but introduces dependency to a very experimental extension). For example, that way one can export the universal sentence encoder (including tokenizer) to ONNX. Example here. I cannot provide anything useful here.
The text was updated successfully, but these errors were encountered: