COMPAT: 0.18.0 scipy dev changes causing interpolation failures

[jmwoloso]

Code Sample, a copy-pastable example if possible

python -c 'import pandas;pandas.test()'

Expected Output

No errors/failures

output of pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.4.3.final.0
python-bits: 64
OS: Linux
OS-release: 3.19.0-32-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.18.0
nose: 1.3.7
pip: 8.1.1
setuptools: 20.7.0
Cython: 0.23.4
numpy: 1.12.0.dev0+47b6c2b
scipy: 0.18.0.dev0+609facc
statsmodels: None
xarray: None
IPython: 4.1.1
sphinx: 1.3.5
patsy: 0.4.0
dateutil: 2.4.2
pytz: 2015.7
blosc: 1.2.8
bottleneck: 1.0.0
tables: None
numexpr: 2.5.2
matplotlib: 1.5.1
openpyxl: 2.3.5
xlrd: 0.9.4
xlwt: 1.0.0
xlsxwriter: 0.7.7
lxml: 3.4.4
bs4: 4.4.1
html5lib: 0.9999999
httplib2: 0.9.2
apiclient: 1.4.2
sqlalchemy: 1.0.9
pymysql: 0.6.7.None
psycopg2: None
jinja2: 2.8
boto: 2.39.0

Hello All,

Can anyone provide any insight into these errors and failures (see attachment)
pandas_test.txt

[jreback]

pls try with pandas master

though almost all of these are I think some changes in scipy

[jmwoloso]

Ok, cool. I'll try that now. Thank you! FYI and for anyone else that might read this, the NumPy and SciPy dev versions that I have installed (see above) actually pass all the tests on my Linux box with MKL and the -O3 compiler setting (a first in my experience!). Hence the reason I'm using them. I'll report back in a few about the Master build.

[jmwoloso]

Here are the results using the Master build (ouch!) I'll down grade to the -O1 flag for NumPy/SciPy if I need to. Pandas is too critical to my work. Really, SciPy is the only one that gives issues now with the -O3 setting. NumPy always builds fine and Pandas usually does as well. Any further suggestions? I've attached the results. Thanks for your help. I appreciate it.
pandas_tests.txt

[jreback]

I think you are having some compiling issues as you are failing on msgpack/sparse things. We don't have any failures with numpy master (except 1 known ATM). We use scipy latest released. If they have changed things it may have broken something (looks like some changes in interpolation).

Why would you compile & run from unreleased versions? this is very very odd.

[jreback]

@TomAugspurger if you'd have a look at the interpolation issues would be great. And maybe see what actually happened.

[jmwoloso]

Well, I was experimenting/attempting to get the -O3 flag to work for the MKL. Pandas 0.17.0 plus these versions of NumPy & SciPy worked on my system before.

[jreback]

ok I have a branch that tests from scipy master now: https://travis-ci.org/jreback/pandas/jobs/122835131

@jmwoloso if you could look and see what scipy is changing would be great thanks!

[jreback]

ok so now the 3.5-dev build has scipy installed here

14 failures from scipy (which is probably all of them). obviously something moved around. So we should push back that this is not compat.

[jmwoloso]

@jreback The date of the last SciPy master I cloned was February 11th with the commit for that happening the day before on the 10th. The commit hash was 609facc07ae8437a8480157cc6da500d929eb00c so quite a bit has changed since then. Unfortunately, the last SciPy version I had before that was 0.15.1. Should I track down the last appearance of that in SciPy or should we wait to hear from the SciPy folks regarding the issue you opened (or both)?

EDIT: And my apologies, I was away from home most of the day or I would have gotten back to you sooner.

[jreback]

it appears that function was deprecated a while back - we really don't see the deprecation notices

so need a fix to provide compat with old and new methods

should be straightforward to do

[TomAugspurger]

@jmwoloso are you able to submit a PR for this? The methods are here and some documentation to fix here. Probably need to remove it from the tests as well.

[ev-br]

Like I was saying in the scipy issue, BPoly.from_derivatives was cooked up specifically as a replacement for the PiecewisePolynomial.

In [39]: x = np.arange(11, dtype=float)

In [40]: y = x**2

In [41]: y = y.reshape(-1, 1)

In [42]: p = PiecewisePolynomial(x, y)

In [43]: xx = (x[1:] + x[:-1]) /2

In [44]: p(xx)
Out[44]: array([  0.5,   2.5,   6.5,  12.5,  20.5,  30.5,  42.5,  56.5,  72.5,  90.5])

In [45]: from scipy.interpolate import BPoly

In [46]: b = BPoly.from<TAB>
BPoly.from_derivatives  BPoly.from_power_basis  

In [46]: b = BPoly.from_derivatives(x, y)

In [47]: b(xx)
Out[47]: array([  0.5,   2.5,   6.5,  12.5,  20.5,  30.5,  42.5,  56.5,  72.5,  90.5])

[jreback]

thanks @ev-br yes that's what I would do here. detect scipy version and go with old function < 0.18 and new after.

[jmwoloso]

@TomAugspurger Sure, I'll give it a shot!

[jmwoloso]

@TomAugspurger Quick question, I'm just removing piecewise_polynomial altogether correct? Or am I replacing it with BPoly.from_derivatives (suggested in scipy/scipy#6063 and above by @ev-br) in the list in pandas.core.missing and from the docs in pandas.core.generic?

[jreback]

you need to place conditionally depending on the version of scipy

[TomAugspurger]

So you'll check the scripy version and raise and error of it's too old. The tests should be similar, skipping if the version is too old.

On Apr 23, 2016, at 18:43, Jeff Reback notifications@github.com wrote:

you need to place conditionally depending on the version of scipy

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

[jmwoloso]

@jreback @TomAugspurger Ok, perfect.

[jreback]

no I would simply switch the function for newer versions

for scipy <= 0.17 it should be unchanged

[jmwoloso]

@jreback Will do.

[jreback]

@jmwoloso if you could do this would be great.

[jmwoloso]

Yeah, you can assign it to me if you'd like. I've forked the repo and got everything set up to contribute (this is my first time with any project) and now I'm just trying to get a lay of the land with the code I'll be changing so I understand what is doing what, if that makes sense?

[jreback]

@jmwoloso great!. contributing docs are here. Just trying to get 0.18.1 done soon.

[jmwoloso]

@jreback Awesome. Yeah, I checked out the contributing docs already, they do a great job of explaining how to set everything up to help out. Working on it now, I'll post questions here if I have any.

[jmwoloso]

@jreback @TomAugspurger Apologies, I thought your responses to the pull request would show up here as well, but they were just going to my mailing list folder. I'm working on your suggestions now and will push an update once I'm done. Thank you for the feedback thus far!

[jreback]

@jmwoloso they show up on whatever issue we were writing them on, in this case the PR itself.