Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate TF-probablity performance issue #1075

Closed
jerryyin opened this issue Aug 7, 2020 · 0 comments
Closed

Investigate TF-probablity performance issue #1075

jerryyin opened this issue Aug 7, 2020 · 0 comments

Comments

@jerryyin
Copy link
Member

jerryyin commented Aug 7, 2020

The context of it is related with TF-issue#954. There are two tasks related with the issue:

  1. Need to understand why ROCm XLA produce the warning "Unexpectedly high number of iterations...". Under the hood this seems to be indicating the pass always changes the hlo code and never converges. I have recorded that HloPassFix has been running on following passes: simplification, algsimp, fusion. Some improper implementation should have affected the ROCm platoform. However, this should only affect compilation behavior, not runtime behavior. (Maybe functional behavior if the optimization is done wrong.)
  2. The performance issue, which according to the user narrowed down to NoUTurnSampler. By running with test_nuts.py I can confirm that the GPU (non-XLA) is running slower than CPU too. That gives a baseline script to test with.
@jerryyin jerryyin closed this as completed Aug 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant