New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI numerical issue #2388

Closed
liwchang opened this Issue Jan 8, 2019 · 6 comments

Comments

Projects
None yet
2 participants
@liwchang
Copy link
Contributor

liwchang commented Jan 8, 2019

CI seems too numerically sensitive.
Also, tvm using random numbers in unit tests made it even worse.
That caused too many false alarms during CI.

Maybe fixing seed or inputs can resolve this issue.

-L

@tqchen tqchen closed this Jan 8, 2019

@tqchen tqchen reopened this Jan 8, 2019

@tqchen

This comment has been minimized.

Copy link
Member

tqchen commented Jan 8, 2019

So far there is not too many flaky cases, if there are, we should fix the test-case to relax the bound, and nail down the flay testcases

@liwchang

This comment has been minimized.

Copy link
Contributor

liwchang commented Jan 8, 2019

Thanks.

I encountered one in logSoftMax today.

Let's keep this thread for a few while.
I think others might wanna report it.

@tqchen

This comment has been minimized.

Copy link
Member

tqchen commented Jan 8, 2019

As per current guideline, https://github.com/dmlc/tvm/blob/master/.github/ISSUE_TEMPLATE.md we try to close issues aggressively and make that sure each issue is actionable. This approach avoids a huge pile of issues that nobody gets attend to. So I would recommend close this one, open a thread in https://discuss.tvm.ai/ and open separate actionable items as issues, for example:

@liwchang

This comment has been minimized.

Copy link
Contributor

liwchang commented Jan 8, 2019

Sure.
At the host, do you mind creating a thread, so everyone can report to the same one? :)

For an actionable item, I still suggest avoiding using pure random numbers, at least we should fix seeds.

@tqchen

This comment has been minimized.

Copy link
Member

tqchen commented Jan 8, 2019

The random number is useful for test-cases and I believe in most cases, as long as we make sure the ranges are right(correct interval of the distribution for certain functions) or have the right tol value, most cases are not flaky. Using random number can sometimes help us find problems.

@tqchen

This comment has been minimized.

Copy link
Member

tqchen commented Jan 8, 2019

centralized thread created per your suggestion https://discuss.tvm.ai/t/flaky-test-cases-thread/1437

@tqchen tqchen closed this Jan 8, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment