RFC: Bump NumPy version requirement

[larsoner]

SciPy 1.2.0 is coming out now, and the next version (1.3.0) will require NumPy 1.13.3+. If we make a similar jump, we can simplify a few things in our code base, including no need to add another thing to fixes.py in #5754.

We have a lot of places we could use keepdims=True instead of [:, np.newaxis, :], as well (though this might only require NumPy 1.10.0).

NumPy 1.13 came out in June 2017. We are probably going to release 0.18 in May/June 2019, so 1.13 will be ~2 years old by then. Perhaps we should start using "about 2 years old" as a guide for how far back we need to support our dependencies?

[massich]

Perhaps we should start using "about 2 years old" as a guide for how far back we need to support our dependencies?

+1e16

[mmagnuski]

+10

[agramfort]

what version do you have on ubuntu stable? I think it's a good target. It should work with stable linux using basic dependencies (numpy, scipy, matplotlib)

[larsoner]

• released 0.5 years ago, 18.04 runs 1.13
• released 2.5 years ago, 16.04 runs 1.11
• released 4.5 years ago, 14.04 (EOL 4/2019) runs 1.8

FYI matplotlib targets just the last 3 years of dependency (NumPy) releases, which today would mean 1.10.0. Because of the Py3k transition (which we are making in 0.18), SciPy is going to require NumPy 1.13.3+ (but in general I think targets 2 or 3 years as well?).

Ubuntu LTS support -- at least 16.04 or older -- will thus end up making us more restrictive than scipy (but about the same as matplotlib) for our new features. Now that these things are all 1) pip-wheel-installable (with OpenBLAS) and 2) conda installable, this seems like a less pressing issue than it would have been, say, a few years ago. So my thinking has changed to be less restrictive...

[agramfort]

nilearn is requiring: - Numpy >= 1.11 https://github.com/nilearn/nilearn

…

[larsoner]

Knowing who runs nilearn and who you hang out with, I'd bet the logic is the same :) i.e., pinning to the stable Linux release.

FWIW Thomas Caswell of matplotlib makes some good points about why this is possibly an outdated view of compatibility in the Python ecosystem over on the SciPy PR, but the most relevant bits are:

Given the rise of user-space python I am less concerned about staying compatible with the version of python shipped by the stable linux distributions. ... If users want to user newer versions of our stuff they should be using some sort of user-space enviroment management (as sudo pip install will eventually end in tears) so it does not seem too big of a lift to use one that also manages python versions.

I like the time-based policy because it is not subjective (don't have to define a 'major' linux distribution, don't have to decide when 'enough' of them have moved, etc) and it easy to compute ...

Formerly I would have agreed with the old-linux pinning. But I tend to see the time-based system as a better fit nowadays.

[cbrnr]

+1 to what @larsoner said and +1 to bumping the version.

[agramfort]

no interaction with debian packaging? see recent comment. Honestly I don't think it's such a pain point to support old numpy. I tend to write numpy code which works on very old versions but maybe I am just old :)

[larsoner]

no interaction with debian packaging? see recent comment.

Not that I see. Assuming the blanket policy is not to bump versions during the support window, if the Debian people are packaging NumPy 1.11 then they shouldn't be packaging MNE 0.17/0.18 but rather 0.12 with it (which was released around the same time in spring 2016). Otherwise they are bumping versions of some packages and not others. But maybe I don't understand how they freeze or upgrade versions.

I tend to write numpy code which works on very old versions but maybe I am
just old :)

Or maybe just not writing enough code :) In the last week:

• @mmagnuski needed np.unique with an axis arg in [MRG] ENH: alternative layout of extra points in interpolation #5754, so he had to create a workaround in fixes.py.
• Yesterday I wrote MRG: Add helper functions for label data #5805, where I put in a SciPy 0.19+ requirement because csr_matrix.argmax did not exist before 0.19+. The naive workaround, which is to loop over the rows to do this operation, is an order of magnitude slower. I thought that the code to do it efficiently would be in Cython but it's not, so I can at least also put it in fixes.py.

And looking forward:

• A year ago I added scipy.signal.window.dpss to compute DPSS windows, and it was released in SciPy 1.1.0 (it should be faster and it would simplify our code base to use it!). 1.1.0 will first appear in an LTS in 2020, which will be "2 releases back", i.e. usable by us, 3.5 years from now (April 2022). We can't backport this function because it needs a LAPACK wrapper that was introduced with it. The proposed "2 years" policy would allow us to use it in 1.5.

In the meantime, though, I'll write a wrapper that chooses what to use, just like I've done for sosfilt and sosfiltfilt and other related filtering functions.

I get the sense that you will not budge on this for now @agramfort so I'll close, but I think we should revisit this issue if the SciPy and matplotlib (and other) developers converge on a common policy. I don't see value in deviating from the scientific-community-standard policy if it gets established as being 2- or 3-years back. Would you agree on that?

[agramfort]

sorry to be a pain but I'd like us to move as fast as the common denominator of our ecosystem. scikit-learn - NumPy (>= 1.11.0) - SciPy (>= 0.17.0) it can be a trade off but I don't see a huge maintenance burden by supporting older numpy so i'd rather not reduce our impact if the cost is limited. my 2c

[cbrnr]

I would lean towards requiring new versions. As @larsoner said, using old version holds us back and requires us to code workarounds. I don't know of a single person who has to use an ancient platform for scientific Python (but that may just be a coincidence). However, given that scientific Python runs in userland (well, that's the wrong term, what I mean is that no admin privileges are required), I don't see how anyone could not upgrade their package versions. We don't have to be on the cutting edge, but using anything older than 3 years seems a bit odd.

[agramfort]

I propose to resurrect this conversation in 6 months.