New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

COMPAT: 0.18.0 scipy dev changes causing interpolation failures #12887

Closed
jmwoloso opened this Issue Apr 13, 2016 · 26 comments

Comments

Projects
None yet
4 participants
@jmwoloso
Contributor

jmwoloso commented Apr 13, 2016

Code Sample, a copy-pastable example if possible

python -c 'import pandas;pandas.test()'

Expected Output

No errors/failures

output of pd.show_versions()


INSTALLED VERSIONS
------------------
commit: None
python: 3.4.3.final.0
python-bits: 64
OS: Linux
OS-release: 3.19.0-32-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.18.0
nose: 1.3.7
pip: 8.1.1
setuptools: 20.7.0
Cython: 0.23.4
numpy: 1.12.0.dev0+47b6c2b
scipy: 0.18.0.dev0+609facc
statsmodels: None
xarray: None
IPython: 4.1.1
sphinx: 1.3.5
patsy: 0.4.0
dateutil: 2.4.2
pytz: 2015.7
blosc: 1.2.8
bottleneck: 1.0.0
tables: None
numexpr: 2.5.2
matplotlib: 1.5.1
openpyxl: 2.3.5
xlrd: 0.9.4
xlwt: 1.0.0
xlsxwriter: 0.7.7
lxml: 3.4.4
bs4: 4.4.1
html5lib: 0.9999999
httplib2: 0.9.2
apiclient: 1.4.2
sqlalchemy: 1.0.9
pymysql: 0.6.7.None
psycopg2: None
jinja2: 2.8
boto: 2.39.0

Hello All,

Can anyone provide any insight into these errors and failures (see attachment)
pandas_test.txt

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Apr 13, 2016

Contributor

pls try with pandas master

though almost all of these are I think some changes in scipy

Contributor

jreback commented Apr 13, 2016

pls try with pandas master

though almost all of these are I think some changes in scipy

@jmwoloso

This comment has been minimized.

Show comment
Hide comment
@jmwoloso

jmwoloso Apr 13, 2016

Contributor

Ok, cool. I'll try that now. Thank you! FYI and for anyone else that might read this, the NumPy and SciPy dev versions that I have installed (see above) actually pass all the tests on my Linux box with MKL and the -O3 compiler setting (a first in my experience!). Hence the reason I'm using them. I'll report back in a few about the Master build.

Contributor

jmwoloso commented Apr 13, 2016

Ok, cool. I'll try that now. Thank you! FYI and for anyone else that might read this, the NumPy and SciPy dev versions that I have installed (see above) actually pass all the tests on my Linux box with MKL and the -O3 compiler setting (a first in my experience!). Hence the reason I'm using them. I'll report back in a few about the Master build.

@jmwoloso

This comment has been minimized.

Show comment
Hide comment
@jmwoloso

jmwoloso Apr 13, 2016

Contributor

Here are the results using the Master build (ouch!) I'll down grade to the -O1 flag for NumPy/SciPy if I need to. Pandas is too critical to my work. Really, SciPy is the only one that gives issues now with the -O3 setting. NumPy always builds fine and Pandas usually does as well. Any further suggestions? I've attached the results. Thanks for your help. I appreciate it.
pandas_tests.txt

Contributor

jmwoloso commented Apr 13, 2016

Here are the results using the Master build (ouch!) I'll down grade to the -O1 flag for NumPy/SciPy if I need to. Pandas is too critical to my work. Really, SciPy is the only one that gives issues now with the -O3 setting. NumPy always builds fine and Pandas usually does as well. Any further suggestions? I've attached the results. Thanks for your help. I appreciate it.
pandas_tests.txt

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Apr 13, 2016

Contributor

I think you are having some compiling issues as you are failing on msgpack/sparse things. We don't have any failures with numpy master (except 1 known ATM). We use scipy latest released. If they have changed things it may have broken something (looks like some changes in interpolation).

Why would you compile & run from unreleased versions? this is very very odd.

Contributor

jreback commented Apr 13, 2016

I think you are having some compiling issues as you are failing on msgpack/sparse things. We don't have any failures with numpy master (except 1 known ATM). We use scipy latest released. If they have changed things it may have broken something (looks like some changes in interpolation).

Why would you compile & run from unreleased versions? this is very very odd.

@jreback jreback added the Testing label Apr 13, 2016

@jreback jreback changed the title from Errors & Failures Pandas 0.18.0 to COMPAT: 0.18.0 scipy dev changes causing interpolation failures Apr 13, 2016

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Apr 13, 2016

Contributor

@TomAugspurger if you'd have a look at the interpolation issues would be great. And maybe see what actually happened.

Contributor

jreback commented Apr 13, 2016

@TomAugspurger if you'd have a look at the interpolation issues would be great. And maybe see what actually happened.

@jmwoloso

This comment has been minimized.

Show comment
Hide comment
@jmwoloso

jmwoloso Apr 13, 2016

Contributor

Well, I was experimenting/attempting to get the -O3 flag to work for the MKL. Pandas 0.17.0 plus these versions of NumPy & SciPy worked on my system before.

Contributor

jmwoloso commented Apr 13, 2016

Well, I was experimenting/attempting to get the -O3 flag to work for the MKL. Pandas 0.17.0 plus these versions of NumPy & SciPy worked on my system before.

@jreback jreback added this to the 0.18.1 milestone Apr 13, 2016

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Apr 13, 2016

Contributor

ok I have a branch that tests from scipy master now: https://travis-ci.org/jreback/pandas/jobs/122835131

@jmwoloso if you could look and see what scipy is changing would be great thanks!

Contributor

jreback commented Apr 13, 2016

ok I have a branch that tests from scipy master now: https://travis-ci.org/jreback/pandas/jobs/122835131

@jmwoloso if you could look and see what scipy is changing would be great thanks!

jreback added a commit to jreback/pandas that referenced this issue Apr 13, 2016

TST: remove ResourceWarnings from stat by auto-closing iterator
TST: install scipy wheels from master on 3.5/dev, xref pandas-dev#12887

TST: skip writing excel if no writers installed
@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Apr 14, 2016

Contributor

ok so now the 3.5-dev build has scipy installed here

14 failures from scipy (which is probably all of them). obviously something moved around. So we should push back that this is not compat.

Contributor

jreback commented Apr 14, 2016

ok so now the 3.5-dev build has scipy installed here

14 failures from scipy (which is probably all of them). obviously something moved around. So we should push back that this is not compat.

@jmwoloso

This comment has been minimized.

Show comment
Hide comment
@jmwoloso

jmwoloso Apr 14, 2016

Contributor

@jreback The date of the last SciPy master I cloned was February 11th with the commit for that happening the day before on the 10th. The commit hash was 609facc07ae8437a8480157cc6da500d929eb00c so quite a bit has changed since then. Unfortunately, the last SciPy version I had before that was 0.15.1. Should I track down the last appearance of that in SciPy or should we wait to hear from the SciPy folks regarding the issue you opened (or both)?

EDIT: And my apologies, I was away from home most of the day or I would have gotten back to you sooner.

Contributor

jmwoloso commented Apr 14, 2016

@jreback The date of the last SciPy master I cloned was February 11th with the commit for that happening the day before on the 10th. The commit hash was 609facc07ae8437a8480157cc6da500d929eb00c so quite a bit has changed since then. Unfortunately, the last SciPy version I had before that was 0.15.1. Should I track down the last appearance of that in SciPy or should we wait to hear from the SciPy folks regarding the issue you opened (or both)?

EDIT: And my apologies, I was away from home most of the day or I would have gotten back to you sooner.

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Apr 14, 2016

Contributor

it appears that function was deprecated a while back - we really don't see the deprecation notices

so need a fix to provide compat with old and new methods

should be straightforward to do

Contributor

jreback commented Apr 14, 2016

it appears that function was deprecated a while back - we really don't see the deprecation notices

so need a fix to provide compat with old and new methods

should be straightforward to do

@TomAugspurger

This comment has been minimized.

Show comment
Hide comment
@TomAugspurger

TomAugspurger Apr 14, 2016

Contributor

@jmwoloso are you able to submit a PR for this? The methods are here and some documentation to fix here. Probably need to remove it from the tests as well.

Contributor

TomAugspurger commented Apr 14, 2016

@jmwoloso are you able to submit a PR for this? The methods are here and some documentation to fix here. Probably need to remove it from the tests as well.

@ev-br

This comment has been minimized.

Show comment
Hide comment
@ev-br

ev-br Apr 14, 2016

Like I was saying in the scipy issue, BPoly.from_derivatives was cooked up specifically as a replacement for the PiecewisePolynomial.

In [39]: x = np.arange(11, dtype=float)

In [40]: y = x**2

In [41]: y = y.reshape(-1, 1)

In [42]: p = PiecewisePolynomial(x, y)

In [43]: xx = (x[1:] + x[:-1]) /2

In [44]: p(xx)
Out[44]: array([  0.5,   2.5,   6.5,  12.5,  20.5,  30.5,  42.5,  56.5,  72.5,  90.5])

In [45]: from scipy.interpolate import BPoly

In [46]: b = BPoly.from<TAB>
BPoly.from_derivatives  BPoly.from_power_basis  

In [46]: b = BPoly.from_derivatives(x, y)

In [47]: b(xx)
Out[47]: array([  0.5,   2.5,   6.5,  12.5,  20.5,  30.5,  42.5,  56.5,  72.5,  90.5])

ev-br commented Apr 14, 2016

Like I was saying in the scipy issue, BPoly.from_derivatives was cooked up specifically as a replacement for the PiecewisePolynomial.

In [39]: x = np.arange(11, dtype=float)

In [40]: y = x**2

In [41]: y = y.reshape(-1, 1)

In [42]: p = PiecewisePolynomial(x, y)

In [43]: xx = (x[1:] + x[:-1]) /2

In [44]: p(xx)
Out[44]: array([  0.5,   2.5,   6.5,  12.5,  20.5,  30.5,  42.5,  56.5,  72.5,  90.5])

In [45]: from scipy.interpolate import BPoly

In [46]: b = BPoly.from<TAB>
BPoly.from_derivatives  BPoly.from_power_basis  

In [46]: b = BPoly.from_derivatives(x, y)

In [47]: b(xx)
Out[47]: array([  0.5,   2.5,   6.5,  12.5,  20.5,  30.5,  42.5,  56.5,  72.5,  90.5])
@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Apr 14, 2016

Contributor

thanks @ev-br yes that's what I would do here. detect scipy version and go with old function < 0.18 and new after.

Contributor

jreback commented Apr 14, 2016

thanks @ev-br yes that's what I would do here. detect scipy version and go with old function < 0.18 and new after.

@jmwoloso

This comment has been minimized.

Show comment
Hide comment
@jmwoloso

jmwoloso Apr 14, 2016

Contributor

@TomAugspurger Sure, I'll give it a shot!

Contributor

jmwoloso commented Apr 14, 2016

@TomAugspurger Sure, I'll give it a shot!

@jmwoloso

This comment has been minimized.

Show comment
Hide comment
@jmwoloso

jmwoloso Apr 23, 2016

Contributor

@TomAugspurger Quick question, I'm just removing piecewise_polynomial altogether correct? Or am I replacing it with BPoly.from_derivatives (suggested in scipy/scipy#6063 and above by @ev-br) in the list in pandas.core.missing and from the docs in pandas.core.generic?

Contributor

jmwoloso commented Apr 23, 2016

@TomAugspurger Quick question, I'm just removing piecewise_polynomial altogether correct? Or am I replacing it with BPoly.from_derivatives (suggested in scipy/scipy#6063 and above by @ev-br) in the list in pandas.core.missing and from the docs in pandas.core.generic?

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Apr 23, 2016

Contributor

you need to place conditionally depending on the version of scipy

Contributor

jreback commented Apr 23, 2016

you need to place conditionally depending on the version of scipy

@TomAugspurger

This comment has been minimized.

Show comment
Hide comment
@TomAugspurger

TomAugspurger Apr 24, 2016

Contributor

So you'll check the scripy version and raise and error of it's too old. The tests should be similar, skipping if the version is too old.

On Apr 23, 2016, at 18:43, Jeff Reback notifications@github.com wrote:

you need to place conditionally depending on the version of scipy


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

Contributor

TomAugspurger commented Apr 24, 2016

So you'll check the scripy version and raise and error of it's too old. The tests should be similar, skipping if the version is too old.

On Apr 23, 2016, at 18:43, Jeff Reback notifications@github.com wrote:

you need to place conditionally depending on the version of scipy


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

@jmwoloso

This comment has been minimized.

Show comment
Hide comment
@jmwoloso

jmwoloso Apr 24, 2016

Contributor

@jreback @TomAugspurger Ok, perfect.

Contributor

jmwoloso commented Apr 24, 2016

@jreback @TomAugspurger Ok, perfect.

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Apr 24, 2016

Contributor

no I would simply switch the function for newer versions

for scipy <= 0.17 it should be unchanged

Contributor

jreback commented Apr 24, 2016

no I would simply switch the function for newer versions

for scipy <= 0.17 it should be unchanged

@jmwoloso

This comment has been minimized.

Show comment
Hide comment
@jmwoloso

jmwoloso Apr 24, 2016

Contributor

@jreback Will do.

Contributor

jmwoloso commented Apr 24, 2016

@jreback Will do.

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Apr 26, 2016

Contributor

@jmwoloso if you could do this would be great.

Contributor

jreback commented Apr 26, 2016

@jmwoloso if you could do this would be great.

@jmwoloso

This comment has been minimized.

Show comment
Hide comment
@jmwoloso

jmwoloso Apr 26, 2016

Contributor

Yeah, you can assign it to me if you'd like. I've forked the repo and got everything set up to contribute (this is my first time with any project) and now I'm just trying to get a lay of the land with the code I'll be changing so I understand what is doing what, if that makes sense?

Contributor

jmwoloso commented Apr 26, 2016

Yeah, you can assign it to me if you'd like. I've forked the repo and got everything set up to contribute (this is my first time with any project) and now I'm just trying to get a lay of the land with the code I'll be changing so I understand what is doing what, if that makes sense?

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Apr 26, 2016

Contributor

@jmwoloso great!. contributing docs are here. Just trying to get 0.18.1 done soon.

Contributor

jreback commented Apr 26, 2016

@jmwoloso great!. contributing docs are here. Just trying to get 0.18.1 done soon.

@jmwoloso

This comment has been minimized.

Show comment
Hide comment
@jmwoloso

jmwoloso Apr 26, 2016

Contributor

@jreback Awesome. Yeah, I checked out the contributing docs already, they do a great job of explaining how to set everything up to help out. Working on it now, I'll post questions here if I have any.

Contributor

jmwoloso commented Apr 26, 2016

@jreback Awesome. Yeah, I checked out the contributing docs already, they do a great job of explaining how to set everything up to help out. Working on it now, I'll post questions here if I have any.

@jmwoloso

This comment has been minimized.

Show comment
Hide comment
@jmwoloso

jmwoloso Apr 27, 2016

Contributor

@jreback @TomAugspurger Apologies, I thought your responses to the pull request would show up here as well, but they were just going to my mailing list folder. I'm working on your suggestions now and will push an update once I'm done. Thank you for the feedback thus far!

Contributor

jmwoloso commented Apr 27, 2016

@jreback @TomAugspurger Apologies, I thought your responses to the pull request would show up here as well, but they were just going to my mailing list folder. I'm working on your suggestions now and will push an update once I'm done. Thank you for the feedback thus far!

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Apr 27, 2016

Contributor

@jmwoloso they show up on whatever issue we were writing them on, in this case the PR itself.

Contributor

jreback commented Apr 27, 2016

@jmwoloso they show up on whatever issue we were writing them on, in this case the PR itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment