test_n_iter_without_progress fails on osx

[jschueller]

The test sklearn.manifold.tests.test_t_sne.test_n_iter_without_progress fails on osx
when building for conda-forge (https://travis-ci.org/conda-forge/scikit-learn-feedstock/jobs/212970171)
It happens with version 0.18.1, regardless of the python version (2.7, 3.5 and 3.6).
On linux the same test succeeds.

======================================================================

FAIL: sklearn.manifold.tests.test_t_sne.test_n_iter_without_progress

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/Users/travis/miniconda3/conda-bld/scikit-learn_1490014369633/_t_env/lib/python2.7/site-packages/nose/case.py", line 197, in runTest

    self.test(*self.arg)

  File "/Users/travis/miniconda3/conda-bld/scikit-learn_1490014369633/_t_env/lib/python2.7/site-packages/sklearn/manifold/tests/test_t_sne.py", line 584, in test_n_iter_without_progress

    "last 2 episodes. Finished.", out)

AssertionError: 'did not make any progress during the last 2 episodes. Finished.' not found in '[t-SNE] Computing pairwise distances...\n[t-SNE] Computed conditional probabilities for sample 100 / 100\n[t-SNE] Mean sigma: 0.765549\n[t-SNE] Iteration 25: error = 13.5941929, gradient norm = 0.0787243\n[t-SNE] Iteration 50: error = 13.3894499, gradient norm = 0.0770499\n[t-SNE] Iteration 75: error = 17.2619819, gradient norm = 0.2986331\n[t-SNE] Iteration 100: error = 16.4814787, gradient norm = 0.0060466\n[t-SNE] KL divergence after 100 iterations with early exaggeration: 16.481479\n[t-SNE] Iteration 125: error = 2.3524479, gradient norm = 0.0015101\n[t-SNE] Iteration 150: error = 2.2885583, gradient norm = 0.0013097\n[t-SNE] Iteration 175: error = 2.2739044, gradient norm = 0.0012655\n[t-SNE] Iteration 200: error = 2.2700430, gradient norm = 0.0012547\n[t-SNE] Iteration 225: error = 2.2689857, gradient norm = 0.0012518\n[t-SNE] Iteration 250: error = 2.2686934, gradient norm = 0.0012510\n[t-SNE] Iteration 275: error = 2.2686124, gradient norm = 0.0012508\n[t-SNE] Iteration 300: error = 2.2685899, gradient norm = 0.0012507\n[t-SNE] Iteration 325: error = 2.2685837, gradient norm = 0.0012507\n[t-SNE] Iteration 350: error = 2.2685820, gradient norm = 0.0012507\n[t-SNE] Iteration 375: error = 2.2685815, gradient norm = 0.0012507\n[t-SNE] Iteration 400: error = 2.2685814, gradient norm = 0.0012507\n[t-SNE] Iteration 425: error = 2.2685813, gradient norm = 0.0012507\n[t-SNE] Iteration 450: error = 2.2685813, gradient norm = 0.0012507\n[t-SNE] Iteration 475: error = 2.2685813, gradient norm = 0.0012507\n[t-SNE] Iteration 500: error = 2.2685813, gradient norm = 0.0012507\n[t-SNE] Iteration 525: error = 2.2685813, gradient norm = 0.0012507\n[t-SNE] Iteration 550: error = 2.2685813, gradient norm = 0.0012507\n[t-SNE] Iteration 575: error = 2.2685813, gradient norm = 0.0012507\n[t-SNE] Iteration 600: error = 2.2685813, gradient norm = 0.0012507\n[t-SNE] Iteration 625: error = 2.2685813, gradient norm = 0.0012507\n[t-SNE] Iteration 650: error = 2.2685813, gradient norm = 0.0012507\n[t-SNE] Iteration 675: error = 2.2685813, gradient norm = 0.0012507\n[t-SNE] Iteration 700: error = 2.2685813, gradient norm = 0.0012507\n[t-SNE] Iteration 725: error = 2.2685813, gradient norm = 0.0012507\n[t-SNE] Iteration 725: error difference 0.000000. Finished.\n[t-SNE] Error after 725 iterations: 16.481479\n'

----------------------------------------------------------------------

[jakirkham]

FWIW we are building on macOS 10.10 using XCode 6.4 when seeing this.

[lesteve]

I can reproduce on OSX in a conda environment using conda-forge. Note that I don't have the problem in a conda environment not using conda-forge, neither with mkl nor with nomkl (the latter uses Accelerate). From a quick investigation, it seems the cython code in TSNE is not used in this test and only numpy may be to blame. Disclaimer: I haven't tested different numpy versions.

[jschueller]

FYI the bug happens at least with numpy 1.12.1 (from conda-forge), I did not test other numpy versions.

[lesteve]

I tried numpy 1.10 and 1.11 and was able to reproduce the test error with both, so it does not seem related to a particular numpy version.

To sum up the problem only happens on OSX when numpy uses OpenBLAS.

I think this is one of this painful ones where the only thing to do is to track where the first difference happens to eventually manage to put together a stand-alone snippet reproducing the problem with only numpy as dependency.

[lesteve]

It may be worth to retry on master once #8768 has been merged.

[lesteve]

This is fixed now that #8768 has been merged.