Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FIX: Precision on test_ccipcanode_v2. #54

Merged
merged 1 commit into from
Aug 12, 2019
Merged

FIX: Precision on test_ccipcanode_v2. #54

merged 1 commit into from
Aug 12, 2019

Conversation

NiMlr
Copy link
Member

@NiMlr NiMlr commented Aug 12, 2019

The precision on this test seems to be set to sharp, as it keeps failing.

The precision on this test seems to be set to sharp, as it keeps failing.
@NiMlr
Copy link
Member Author

NiMlr commented Aug 12, 2019

I did a test using the repo at this point in time. Namely, I ran test_ccipcanode_v2 400 times. It failed 9 times. Lets assume the repo had only this test. As we currently have 26 jobs running in Travis and given they are independent, to reach a 0.8 probability of a success of a travis run, we would need to have a probability of a single instance of this test succeeding of at least

Assuming we have a probability of the test succeeding of at least that, the probability of observing 9 instances or more out of 400 failing is smaller than

This suggests a more general guideline should exists for the randomized tests, especially considering that we have multiple independent randomized tests, not only this one. This fix does not live up to these standards. However, it fixes the errors for now.

@Stewori
Copy link
Member

Stewori commented Aug 12, 2019

This time test_inverse failed. Still I will check this in now in the hope that no random test fails again. Keep fingers crossed...
Nice math btw. We should pin your thoughts as a separate issue.

@Stewori Stewori merged commit e135f0d into mdp-toolkit:master Aug 12, 2019
@coveralls
Copy link

Coverage Status

Coverage decreased (-0.7%) to 73.112% when pulling 8167644 on NiMlr:patch-2 into 9d2d1be on mdp-toolkit:master.

@NiMlr NiMlr mentioned this pull request Dec 13, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants