Compatibility for Scipy 16.1

[ljwolf]

As @sjsrey found, upgrading to scipy 16.1 in our testing suite causes quite a few errors. The scipy changelog shows that nothing big changed.

• bugfix for ckdtree
• ckdtree distance computation fix

Not sure what, exactly, we need to fix yet, but tests were failing all over weights and esda.

My hope is that the issue with the changes to ckdtree or something slightly different in kdtree.

[sjsrey]

I think for the 1.11 release we pin. Then post release we work on revised tests to reflect the scipy-induced breakage.

[sjsrey]

I think the scipy changes have also induced this bug http://gis.stackexchange.com/questions/181658/distance-based-spatial-weights-in-pysal

[MridulS]

@sjsrey the bug was in scipy right? scipy/scipy#5088 fixes it. As pysal depends on scipy it was carried over to pysal.

[ljwolf]

We still have test failures when we let numpy and scipy float to 1.10.4 and 0.17.0. If anyone has time to look into this more closely, pinpointing the exact combinations of scipy/numpy verisons where this starts breaking could be done using a travis build matrix.

I think some of this is small differences in rounding (like the SUR test failures), but there are substantive differences in how post-16.2 scipy does self-weights. It shows up here. At this point, I also can't tell if this is causing various failures in local G statistics. Those are large enough to be worrisome, and don't appear to be related to outstanding bugs in scipy.

So... while these failures might've started during scipy/scipy#5088, we might need to troubleshoot why test_Distance that rely on scipy kdtrees start failing. (Maybe self-neighbor-NA stuff?)

[ljwolf]

@MridulS If you can, I'd definitely configure travis if you're interested in troubleshooting this particular bug 😃

[MridulS]

@ljwolf Sure, will look into it :)

[dfolch]

@MridulS, thanks for checking this out. Another thing to notice is that Euclidean distance kdtrees could be either scipy.spatial.cKDTree or scipy.spatial.KDTree (depending on the user's version of scipy), but Arc distance kdtrees are always based on scipy.spatial.KDTree. So it might be important to keep track of which type of kdtree is the input to some downstream test.

[MridulS]

I ran a build, https://travis-ci.org/MridulS/pysal/builds/111835893.

Currently, pysal builds if scipy == 0.16.0 and numpy >= 1.10.0 (https://travis-ci.org/MridulS/pysal/jobs/111835899) The scipy failure is due to scipy/scipy#5088 . Something changed between numpy 1.9.2 and 1.10.0 (nothing substantive in release notes that could cause this) which is causing the failures (SUR failures) -> https://travis-ci.org/MridulS/pysal/jobs/111835922 (scipy 0.16, numpy 1.9.2)

Will try to look into the post 16 self weights issue now.

[sjsrey]

I think we are close to fixing this. The one issue remaining is that the order of the neighbors is not robust to change in operating systems as on linux we fail but on OS X, this same check passes.

One option would be to change the tests to use sets rather than lists.

[sjsrey]

Fixed with #756