test_module.test_factorization_machine_module has fixed seed that can mask flakiness #11704

szha · 2018-07-13T05:25:08Z

The unit test in title have been using fixed seed to mask flakiness. Suggested action:

Evaluate whether the test is flaky without fixed seed. If not, remove seed. Else move to 2
If test is flaky, determine whether it's an actual uncaught edge case. If so, fix the operator. Else move to 3
If numerical instability is inevitable, adjust tolerance level appropriately.

apeforest · 2018-07-13T20:48:09Z

Thanks for filing this issue. We will investigate this Flaky test

apeforest · 2018-07-31T19:57:32Z

I could reproduce the issue using random seed 1306649843. The issue is accuracy mismatch. Working on it.

apeforest · 2018-08-01T16:01:42Z

apeforest · 2018-08-03T16:37:45Z

Fixed unit test and added training test. PR: #12023

apeforest · 2018-08-04T03:25:17Z

@szha PR is merged. Please verify and close this issue. Thanks!

haojin2 · 2018-08-16T00:29:01Z

@sandeep-krishnamurthy Related PR is merged, should be good to close.

szha added Test Flaky labels Jul 13, 2018

szha added this to To Do in Tests Improvement via automation Jul 13, 2018

apeforest mentioned this issue Aug 3, 2018

[MXNET-770] Fix flaky test: test_factorization_machine_module #12023

Merged

3 tasks

sandeep-krishnamurthy closed this as completed Aug 16, 2018

Tests Improvement automation moved this from To Do to Done Aug 16, 2018

Provide feedback