Loosen separation test tolerance #302
Conversation
For some tests, the difference between the scores produced by Travis' build and my own build is as large as 3.89588121e-04. In practice, we shouldn't care much about differences larger than 10e-3 for separation tasks since it would be very unusual for anyone to pay attention to differences this small when comparing source separation algorithms.
|
LGTM; I think this is related to our previous headaches deriving from BLAS discrepancies across platforms, but I agree that anything past the 3rd (2nd?) decimal place is unreliable in source sep anyway. |
|
Amazingly, this is still not loose enough for the Python 3.4 build: |
|
Not necessarily, but we have just not updated Travis. That this affects 3.4 and not, say, 3.5 or 3.6 is mostly a coincidence I think.
Ok, will do. |
I dropped 3.4 builds on travis (in librosa) for exactly this reason. Since mir_eval doesn't use any fancy features of >=3.5 (except, maybe, implicitly ordered dictionaries?), I don't see much point in keeping the 3.4 test around. |
For some tests, the difference between the scores produced by Travis' build and my own build is as large as 3.89588121e-04. This was causing Travis to fail. In practice, we shouldn't care much about differences larger than 10e-3 for separation tasks since it would be very unusual for anyone to pay attention to differences this small when comparing source separation algorithms.