Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TestFilterDetections test_mini_batch fails sometimes on some TensorFlow versions #155

Closed
willgraf opened this issue Mar 29, 2019 · 3 comments · Fixed by #263
Closed

TestFilterDetections test_mini_batch fails sometimes on some TensorFlow versions #155

willgraf opened this issue Mar 29, 2019 · 3 comments · Fixed by #263

Comments

@willgraf
Copy link
Contributor

On both TensorFlow version 1.11.0 and 1.12.0 the FilterDetections test case test_mini_batch will fail most of the time inside TravisCI. It will pass eventually if re-run enough.

On TensorFlow version 1.11.0, sometimes the test fails its self.assertAllEqual(actual_scores, expected_scores) test due to a correctness problem (the wrong entry is supprussed).

On TensorFlow version 1.12.0 sometimes the tests fails its self.assertAllEqual(actual_scores, expected_scores) test due to a NonMaxSuppression shape issue.

@willgraf willgraf changed the title FilterDetections mini_batch test fails sometimes on some TensorFlow versions TestFilterDetections test_mini_batch fails sometimes on some TensorFlow versions Mar 29, 2019
@willgraf
Copy link
Contributor Author

I think this is caused by #218. However, when score_threshold is passed to the nms function, we see failures in tensorflow 1.8 only. This is likely duplicate, but will leave it for now.

@willgraf
Copy link
Contributor Author

It looks like the successful tests were just lucky, the PR build had failures for 1.11 and 1.12 on both 2.7 and 3.5. All tests passed for python 3.7, but perhaps that is just lucky.

@willgraf
Copy link
Contributor Author

willgraf commented Sep 17, 2019

This issue is NOT related to score_threshold.


TensorFlow 1.12 failure:

>       np.testing.assert_array_equal(a, b, err_msg="\n".join(msgs))
E       AssertionError: 
E       Arrays are not equal
E       
E       not equal where = (array([0]), array([0]))
E       not equal lhs = [0.9]
E       not equal rhs = [1.]
E       Mismatch: 0.167%
E       Max absolute difference: 0.10000002
E       Max relative difference: 0.10000002
E        x: array([[ 0.9, -1. , -1. , -1. , -1. , -1. , -1. , -1. , -1. , -1. , -1. ,
E               -1. , -1. , -1. , -1. , -1. , -1. , -1. , -1. , -1. , -1. , -1. ,
E               -1. , -1. , -1. , -1. , -1. , -1. , -1. , -1. , -1. , -1. , -1. ,...
E        y: array([[ 1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1.,
E               -1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1.,
E               -1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1.,...

TensorFlow 1.11 failure:

>       np.testing.assert_array_equal(a, b, err_msg=msg)
E       AssertionError: 
E       Arrays are not equal
E       
E       Mismatch: 0.167%
E       Max absolute difference: 0.10000002
E       Max relative difference: 0.10000002
E        x: array([[ 1. , -1. , -1. , -1. , -1. , -1. , -1. , -1. , -1. , -1. , -1. ,
E               -1. , -1. , -1. , -1. , -1. , -1. , -1. , -1. , -1. , -1. , -1. ,
E               -1. , -1. , -1. , -1. , -1. , -1. , -1. , -1. , -1. , -1. , -1. ,...
E        y: array([[ 1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1.,
E               -1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1.,
E               -1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1., -1.,...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant