Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

COMPAT: Consider dropping numpy 1.6 support #7711

Closed
cpcloud opened this issue Jul 9, 2014 · 33 comments

Comments

Projects
None yet
6 participants
@cpcloud
Copy link
Member

commented Jul 9, 2014

NumPy 1.6

NumPy 1.6 introduced numerous bugs around datetimes, forcing a plethora of hacks, and causing numerous stress-induced ulcers for an uncountably infinite number of people.

The time for pandas to lose this vestigial organ has come. The main reason for this is that datetime support is just broken, in nearly every sense of the word. Take a look at some of the conversion code in pandas/core/common.py and you'll see what I'm talking about.

I think we've probably reached a point where supporting numpy 1.6 is just going to cause more pain than it's worth. To me it makes no sense to want to keep upgrading pandas while keeping numpy <= 1.6.1. If you want modern pandas, you need to use modern-ish versions of its dependencies.

@cpcloud cpcloud added this to the 0.15.0 milestone Jul 9, 2014

@jreback

This comment has been minimized.

Copy link
Contributor

commented Jul 9, 2014

i think i raised this for #5108, but maybe its time

@jreback jreback added the CI label Jul 9, 2014

@jreback

This comment has been minimized.

Copy link
Contributor

commented Jul 9, 2014

cc @yarikoptic

can you give an indication when major distros moving to 1.7 for their stable? (iirc somewhere jessie is doing this in oct of this year?)

@jtratner

This comment has been minimized.

Copy link
Contributor

commented Jul 10, 2014

To me it makes no sense to want to keep upgrading pandas while keeping numpy <= 1.6.1. If you want modern pandas, you need to use modern-ish versions of its dependencies.

Excellent point. 👍

@yarikoptic

This comment has been minimized.

Copy link
Contributor

commented Jul 10, 2014

here is the current state of numpy:

Ubuntu   1:1.3.0-3build1    http://packages.ubuntu.com/lucid/python-numpy
Ubuntu   1:1.6.1-6ubuntu1   http://packages.ubuntu.com/precise/python-numpy
Ubuntu   1:1.7.1-1ubuntu1   http://packages.ubuntu.com/raring/python-numpy
Ubuntu   1:1.7.1-1ubuntu1   http://packages.ubuntu.com/saucy/python-numpy
Ubuntu   1:1.8.1-1ubuntu1   http://packages.ubuntu.com/trusty/python-numpy
Ubuntu   1:1.8.1-1ubuntu1   http://packages.ubuntu.com/utopic/python-numpy
Debian   1:1.4.1-5          http://packages.debian.org/squeeze/python-numpy
Debian   1:1.6.2-1.2        http://packages.debian.org/wheezy/python-numpy
Debian   1:1.8.1-1+b1       http://packages.debian.org/jessie/python-numpy
Debian   1:1.8.1-1+b2       http://packages.debian.org/sid/python-numpy

so jessie will have the "modern" numpy, but current stable is at 1.6.2 and current pandas still happy on it. It would be great if dropping it could may be postponed after the next pandas release? according to reasoning above current release has broken handling of datetime on wheezy with numpy 1.6.2 -- then "dropping" support of numpy 1.6 in the next release would simply leave people on stable debian with a broken pandas, or have I misread it?

@jreback

This comment has been minimized.

Copy link
Contributor

commented Jul 10, 2014

@yarikoptic you have misread.

pandas 'fixes' the timedelta support on < 1.7 numpy (using pandas operations). even though numpy is hopelessly broken (its much better in 1.8/1.9). In reality we now DO raise on < 1.7 numpy for unsupported ops.

the issue is that we have lots of test that are skipped on numpy < 1.7 and a fair number of ugly work-arounds for these.

I actually don't see dropping support for < 1.7 as much of a problem, because this is ONLY an issue if:

  • you have a fairly old build (i use wheezy so am guilty!)
  • you install/upgrade to current pandas using PIP
  • you don't install numpy but instead use the system installed version

why ANYONE would do this is beyond me. If you are going to install pandas latest, what's the big deal with install numpy current?

@jreback

This comment has been minimized.

Copy link
Contributor

commented Jul 10, 2014

@cpcloud let's separate the 2.6 part of this to a new issue

@dsm054

This comment has been minimized.

Copy link
Contributor

commented Jul 10, 2014

Now that Python has officially blessed virtual environments with the adoption of venv, the canonical response to "but my system only has library version X installed" is now "so?"

+1 in favour of dropping broken (for our use cases) numpy.

@jreback

This comment has been minimized.

Copy link
Contributor

commented Jul 10, 2014

@cpcloud can you put this on the mailing list as well, just to see if we are missing anything here. (and copy numpy-discussion too).

@cpcloud cpcloud added the Community label Jul 10, 2014

@cpcloud cpcloud changed the title COMPAT: Consider dropping 2.6 support and numpy 1.6 support COMPAT: Consider dropping numpy 1.6 support Jul 10, 2014

@cpcloud

This comment has been minimized.

Copy link
Member Author

commented Jul 10, 2014

Sure

@yarikoptic

This comment has been minimized.

Copy link
Contributor

commented Jul 10, 2014

@jreback regarding three bullet points. "you install/upgrade to current pandas using PIP" -- I agree that if someone does it, he is on his own ;) I was primarily concerned to have a reasonably good version within NeuroDebian for wheezy for at least another year (so people have time to upgrade from wheezy to jessie). If you think that 0.14.0+git17-g3849d5d-1~nd70+1 should be fine -- then go ahead with dropping the support. If you feel that it might be worth making a quick point or snapshot release before you drop 1.6.2 numpy support -- might even be better ;-)

@jreback

This comment has been minimized.

Copy link
Contributor

commented Jul 10, 2014

@yarikoptic we are planning on releasing 0.14.1 tomorrow, which includes < 1.7 support. @cpcloud mentioned this for 0.15.0 approx due end of sept 2014. That work?

@yarikoptic

This comment has been minimized.

Copy link
Contributor

commented Jul 10, 2014

Sounds great to me! Thank you guys!

@yarikoptic

This comment has been minimized.

Copy link
Contributor

commented Jul 10, 2014

BTW FWIW and FYI -- if 0.15.0 comes out in Aug and would be polished enough -- it might still make it into jessie which is to start freezing in Sep . Otherwise jessie would come out with 0.14.1.

@cpcloud

This comment has been minimized.

Copy link
Member Author

commented Jul 17, 2014

Mailing list seems to be a bit silent on this one. FWIW conda ships with numpy 1.8.1 and It seemed like a lot of people at SciPy 2014 were using conda. That said, I'm sure there are stragglers out there. I still don't understand why anyone would keep their numpy at <= 1.6.x and continue to follow pandas releases.

@shoyer

This comment has been minimized.

Copy link
Member

commented Jul 17, 2014

I agree, I don't see a need to keep around support for ancient versions of numpy at this point.

@jreback

This comment has been minimized.

Copy link
Contributor

commented Aug 7, 2014

ok, let's do this one. I think its time.

@cpcloud ?

prob need to search the code and remove stuff that is not necessary (and a lot of skipping tests)

so change builds 2.6/2.7_locale to numpy 1.7.0 I guess? (I have to create wheels, but no biggie)

@jreback jreback referenced this issue Aug 7, 2014

Merged

WIP: searchsorted implementation #7447

9 of 9 tasks complete
@cpcloud

This comment has been minimized.

Copy link
Member Author

commented Aug 7, 2014

alrighty, into the rabbit hole i go

@jreback

This comment has been minimized.

Copy link
Contributor

commented Aug 7, 2014

their is actually some support for < 1.6 too (grin _np_version_under1p6)....should go too!

@jreback

This comment has been minimized.

Copy link
Contributor

commented Aug 7, 2014

also need to fix setup too

@cpcloud

This comment has been minimized.

Copy link
Member Author

commented Aug 7, 2014

yep there's quite a bit of acrobatics around dates as u know! i'll gut as mcuh as i can and then we can slowly remove things over time if need be

@cpcloud

This comment has been minimized.

Copy link
Member Author

commented Aug 7, 2014

i put 1.7.0 on 2.6 and 1.7.1 on 2.7_locale

@jreback

This comment has been minimized.

Copy link
Contributor

commented Aug 7, 2014

acrobatics in tseries/timedeltas.py (and tslib.pyx) is where a lot of this is

@cpcloud

This comment has been minimized.

Copy link
Member Author

commented Aug 7, 2014

yep

@jreback

This comment has been minimized.

Copy link
Contributor

commented Aug 7, 2014

ok, i'll make wheels tonight

@cpcloud

This comment has been minimized.

Copy link
Member Author

commented Aug 7, 2014

sounds good

@cpcloud

This comment has been minimized.

Copy link
Member Author

commented Aug 7, 2014

actually the np under 1.7 constants can go too, since we're requiring that that be the case now

@jreback

This comment has been minimized.

Copy link
Contributor

commented Aug 7, 2014

yep, you can prob just take them out, and run the tests and see where it breaks :)

@jreback

This comment has been minimized.

Copy link
Contributor

commented Aug 7, 2014

maybe a nice warning in __init__.py as well (in case someone installs pandas somehow w/o pip and relies on a system numpy < 1.7)

@cpcloud

This comment has been minimized.

Copy link
Member Author

commented Aug 7, 2014

good call

@cpcloud cpcloud self-assigned this Aug 7, 2014

@cpcloud cpcloud added the Compat label Aug 7, 2014

@cpcloud

This comment has been minimized.

Copy link
Member Author

commented Aug 11, 2014

i think a raise would be better than warning even though most things would probably work, that's not a very convincing reason.... prevent foot shooting and all

@jreback

This comment has been minimized.

Copy link
Contributor

commented Aug 12, 2014

awesome!!!

@cpcloud

This comment has been minimized.

Copy link
Member Author

commented Aug 12, 2014

:) thanks whoohoo! no more np16!

@jreback

This comment has been minimized.

Copy link
Contributor

commented Aug 12, 2014

hmm I think that I need to take it out of my windows test builds - yeh!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.