-
Notifications
You must be signed in to change notification settings - Fork 216
[dynamo exporter] Fix dynamic shapes with DynamicCache #1832
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Feel free to merge this before the IR change |
@jambayk I think CI is being weird. Would you help me merge it? |
/azp run |
Azure Pipelines successfully started running 2 pipeline(s). |
onnxscript fusion test failed: https://github.com/microsoft/Olive/blob/main/test/unit_test/passes/onnx/test_onnxscript_fusion.py#L45. How will this test change onnxscript fusion behavior? |
Thanks! It looks like TreeSpec is still wrong. I will need another fix on this. |
Describe your changes
transformers introduces "Cache" objects, which is a custom class that contains kv_cache. However, in the current exporter, this becomes an exception of the pre-condition: input should have the same tree spec as the dynamic_shapes. For example, although DynamicCache is now traceable in exporter, it does not make sense to dynamic_shapes. Therefore, we can't use input tree spec to unflatten dynamic_shapes anymore. This PR deletes that, and simply iterates the input tree spec and turn any tuple to list to overcome the format mismatch caused by config.json.
Checklist before requesting a review
lintrunner -a
(Optional) Issue link