Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ideas : Better random testing #16434

Open
sylee957 opened this issue Mar 25, 2019 · 0 comments
Open

Ideas : Better random testing #16434

sylee957 opened this issue Mar 25, 2019 · 0 comments
Labels
Enhancement Testing Related to the test runner. Do not use for test failures unless it relates to the test runner itself

Comments

@sylee957
Copy link
Member

sylee957 commented Mar 25, 2019

Well, I see some random tests are failing, but they are too difficult to debug.

e.g #16288, #14481

I only understand that RNG changes the state every time it computes, and in that is the case, if our only clue is the random seed, the test may have to follow lengthy steps run in the way exactly how the travis do, only to inspect it.

And also, there is a suspicion that random test may create spurious coverage #16196

I think it may be better to

  1. Make more verbose log for random tests, so it may print the exact expression which had failed the testing.

  2. We may have to categorize random tests, like using @randtest, and all random tests can be collected and run in a separate travis stage. It will be good if there is a pytest equivalent wrapper for that.

@sylee957 sylee957 added Enhancement Testing Related to the test runner. Do not use for test failures unless it relates to the test runner itself labels Mar 25, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancement Testing Related to the test runner. Do not use for test failures unless it relates to the test runner itself
Projects
None yet
Development

No branches or pull requests

1 participant