tighten test_topic_word #3280
tighten test_topic_word #3280
Conversation
Codecov Report
@@ Coverage Diff @@
## develop #3280 +/- ##
===========================================
- Coverage 81.43% 81.41% -0.03%
===========================================
Files 122 122
Lines 21052 21107 +55
===========================================
+ Hits 17144 17184 +40
- Misses 3908 3923 +15
Continue to review full report at Codecov.
|
I don't really understand what you mean by "mutants". But if the tests are tighter now while remaining stable, then that's great. |
Mutants here refers to small changes in source code. They are designed to act as a proxy for potential bug in the source code. I automatically generate them based on test coverage (only generate mutantions on the lines covered by the test). |
|
Oh jesus. That's some next level investigation! Could this be applied to other tests in Gensim? Or even other projects? Sounds really interesting. |
Thank you! Currently we are looking at other tests in Gensim and also other projects. We are doing more local testing to make sure the tightened bound does not make the test flaky as well |
|
Nice. Who are "we"? |
|
We are a group of researchers working on such non-deterministic tests in ML projects, particually on tests with loose bounds |
|
@piskvorky @mpenkov Does it look good to merge? |
|
LGTM, merging. Thank you @austereantelope ! |
Hi,
The test
test_topic_wordintest_ldaseqmodel.pyhas an assertion bound (self.assertAlmostEqual(topics[0][0][1], expected_topic_word[0][1], places=2)) that is too loose. This means potential bug in the code could still pass the original test.To quantify this I conducted some experiments where I generated multiple mutations of the source code under test and ran each mutant and the original code 100 times to build a distribution of their outputs. I used KS-test to find mutants that produced a different distribution from the original and use those mutants as a proxy for bugs that could be introduced. In the graph below I show the distribution of both the original code and also the mutants with a different distribution.
Here we see that the bound of
0.005(same as round to 2 decimal place) is too loose since the original distribution (in orange) is much less than0.005. Furthermore in this graph we can observe that there are many mutants (proxy for bugs) that are below the bound as well and that is undesirable since the test should aim to catch potential bugs in code. I quantify the "bug detection" of this assertion by varying the bound in a trade-off graph below.In this graph, I plot the mutant catch rate (ratio of mutant outputs that fail the test) and the original pass rate (ratio of original output that pass the test). The original bound of
0.005(red dotted line) has a catch rate of 0.16To improve this test, I propose to tighten the bound to
0.0012(the blue dotted line). The new bound has a catch rate of 0.375 (+0.215 increase compare to original) while still has >99 % pass rate (test is not flaky, I ran the updated test 500 times and observed >99 % pass rate). I think this is a good balance between improving the bug-detection ability of the test while keep the flakiness of the test low.Do you guys think this makes sense? Please let me know if this looks good or if you have any other suggestions or questions.
My Environment:
my Gensim Experiment SHA:
6e362663f23967f3c1931e2cb18d3d25f92aabb5