-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[TEST] Conv2dGrad Takes long time to finish #8579
Comments
Followup with @altanh showing that this might be related to a perf regression, we will follow up on this thread |
After analysis we believe this regression is due to a combination of changes in #8486 and also an inefficient loop in the
Proposed SolutionThe second point can be easily rectified by hoisting the forward function evaluation outside the hot loop, as the function doesn't change. In fact it only needs to be evaluated once per device and target combination. This will immediately fix the extreme regression. A PR will be posted soon. Regarding the first point, we would like to move away from relying on hidden global caching for performance as much as possible as this has caused confusion in the past. Thus we will not modify the new interpreter behavior. |
Update: the problem goes a bit deeper with the Interpreter executor. Specifically, the function returned by Possible solutions:
I'll go ahead with option 2 in the hot fix PR for now |
Let us go with 1 and just skip the interpreter executor in the grad evaluation. |
Closing this for now, opened #8601 to track the followup item |
Looking at the test time log, https://ci.tlcpack.ai/job/tvm/job/main/1356/testReport/ctypes.tests.python.relay/test_op_grad_level2/
conv2d grad takes a long time to finish(1h)..
Possible cause: this is likely due to our numerical gradient checker trying to do numerical gradient for all directions which scales linearly to the number of entries.
Possible resolutions:
I would suggest we go with K1 with a few directions instead of the full direction checking.
The text was updated successfully, but these errors were encountered: