MAINT: drop Python 2.7 and 3.4

[rgommers]

Updates all build files, docs, CI config files. Also adds # cython: language_level=3 to get rid of the many warnings from Cython.

Should also solve the issues with building wheels (gh-9574), but let's check that after merge.

[charris]

@tylerjereddy Just a reminder that scipy-wheels master branch will also need an update after this goes in.

[larsoner]

Other than one idea maybe worth investigating, looks good to me

[ev-br]

LGTM.

[larsoner]

@rgommers a practical question, why do we bump the NumPy requirement all the way up to 1.13+? It looks like there are 3.6 wheels for 1.12 on PyPi (but only up to 3.5 for NumPy 1.11).

[ilayn]

If I remember correctly that is the earliest version of NumPy that has the wheels for all platforms available.

[larsoner]

I see win 32/64, linux 686/x86_64, and OSX for Python 3.5 and 3.6 for NumPy 1.12.0 in the pip link I posted above. Do we need others or do these not work properly...?

[tacaswell]

Matplotlib is numpy 1.11+ currently as that was the oldest numpy that supported py3.6 iirc.

For Matplotlib we plan to drop py3.5 support in mpl3.1 (early next year), based on 36 months of support for a given python minor version (see https://matplotlib.org/devel/min_dep_policy.html). We should try to aligen these policies across the ecosystem if possible.

[rgommers]

Python 3.7 requires NumPy >= 1.13.1, and it's nicer to have the same minimum numpy version for each python version. This was in my proposal on the list.

[larsoner]

Makes sense to me, thanks for clarifying.

[rgommers]

For Matplotlib we plan to drop py3.5 support in mpl3.1 (early next year), based on 36 months of support for a given python minor version (see https://matplotlib.org/devel/min_dep_policy.html).

For NumPy and SciPy, it has usually been based on what is being shipped by the major Linux distributions (typically Debian and RHEL being the laggards). That's a lowest common denominator.

We should try to aligen these policies across the ecosystem if possible.

Agreed, that would be useful.

[ilayn]

Should we also remove the from __future__ ... clauses? Feels quite awkward though 😃

[tacaswell]

Given the rise of user-space python I am less concerned about staying compatible with the version of python shipped by the stable linux distributions. As they are stable, I do not expect them to package new versions of the scipy stack for their main line (they tend to stay with what ever was out when they released). If users want to user newer versions of our stuff they should be using some sort of user-space enviroment management (as sudo pip install will eventually end in tears) so it does not seem too big of a lift to use one that also manages python versions.

I like the time-based policy because it is not subjective (don't have to define a 'major' linux distribution, don't have to decide when 'enough' of them have moved, etc) and it easy to compute (well, as easy as estimating a release date and doing date-math ever is ;) ).

Changing the numpy support policy to "the oldest that supports the newest python supported" is interesting.

[rgommers]

Should we also remove the from __future__ ... clauses? Feels quite awkward though

We decided not to do any cleanups yet, because 1.2.x is an LTS release and we want to keep the work for backports as low as possible. So I'd prefer to defer all cleanups for now rather than weigh up for each one if it would hinder backports.

[rgommers]

Given the rise of user-space python I am less concerned about staying compatible with the version of python shipped by the stable linux distributions.

I sympathize with the argument, especially now that conda has become a good option. But I know a number of people disagree, and they may have a point - installing Python from source is not something that many people want to do, and not everyone will want to use conda. So this argument is unlikely to be accepted.

I like this discussion and aiming at standardizing this as much as possible. Will move it to a separate issue to not lose track of this.

Changing the numpy support policy to "the oldest that supports the newest python supported" is interesting.

We can't always do this, because we're trying to keep at least 4 supported numpy versions, to help users in their upgrade paths. So say we'll need numpy 1.17.0 for Python 3.8, then we'll go back to requiring different numpy versions for different Python versions.

[rgommers]

Two approving reviews and no unaddressed comments, so let's get this in. Thanks all.

[larsoner]

Thanks @rgommers

[charris]

@rgommers This broke the wheels builds on linux, see https://travis-ci.org/MacPython/scipy-wheels/jobs/471425652. NumPy handles the python_requires differently, but I don't know what the right thing to do for SciPy is.

EDIT: Works on Windows.

NVM, the 2.7 wheels are still being built on scipy-wheels master.

[charris]

I'm trying to update scipy-wheels builds, currently broken. What should the minimum NUMPY_BUILD_DEP be? The minimum NUMPY_TEST_DEP ? Should these both default to 1.13.3?

[rgommers]

Thanks @charris. Yes for master both should be 1.13.3. For 1.2.x there should not be any changes compared to 1.1.1

[charris]

Looks like SciPy built against earlier NumPy versions (1.13.1) but tested with later versions (1.15.4) emits several thousands of PendingDeprecationWarnings. Not sure why that is. Fixed in scipy-wheels by specifying 1.13.3 so the first version bigger than 1.13.1 (cached?) isn't pulled in for the testing, but the problem is probably still there. Should be tested locally just to see.

[charris]

@rgommers What Mac version? Currently scipy-wheels is using 10.6, should it be 10.9?

EDIT: The available Mac libraries for OpenBLAS v0.3.4+ are for 10.9. @matthew-brett Ping.

[rgommers]

yes 10.9 sounds right

[charris]

@rgommers OK, I'll put that in scipy-wheels master. What about the v1.2.x? Scipy 1.2.0 came out with the old OpenBLAS library, and it also has an svd bug, #9620. Let's bring @tylerjereddy in also. I'll make a PR for that and leave the decision up to others.

[rgommers]

Yes I think 1.2.x can also be updated to 10.9.

That SVD bug isn't pretty ....

[jakirkham]

Would it be worth adding SciPy to Python 3 Statement (as NumPy was)?

[rgommers]

I opened an issue for that a while ago: python3statement/python3statement.github.io#158

[jakirkham]

Thanks for the update.