DEPS: Bump numpy to 1.13.3 #25554
DEPS: Bump numpy to 1.13.3 #25554
Conversation
Not due to Cython issues, just to reduce the maintenance burden. See https://mail.python.org/pipermail/scipy-dev/2018-November/023181.html for rationale. +1 for going to 1.13.3 as well. Matplotlib is doing a similar thing (actually they want to be a bit more agressive), and wants to standardize a common Python/NumPy version support policy for core packages (see scipy/scipy#9610, we'll come back on that soon). |
|
Line 227 in 615fbb3 |
Codecov Report
@@ Coverage Diff @@
## master #25554 +/- ##
==========================================
- Coverage 91.75% 91.26% -0.49%
==========================================
Files 173 173
Lines 52960 52961 +1
==========================================
- Hits 48595 48337 -258
- Misses 4365 4624 +259
Continue to review full report at Codecov.
|
Codecov Report
@@ Coverage Diff @@
## master #25554 +/- ##
==========================================
+ Coverage 91.53% 91.55% +0.02%
==========================================
Files 175 175
Lines 52808 52775 -33
==========================================
- Hits 48338 48320 -18
+ Misses 4470 4455 -15
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
@jreback
Regarding pins, please have a look at the OP. Bumping numpy is always a good time to look at the distribution in the CI.
For numpy, there's currently 3x 1.12, 2x 1.13, 1x 1.14, ZERO 1.15 and 7 unpinned.
I'm proposing to make this 3x 1.13, 2x 1.14, 2x 1.15 and 6 unpinned.
For matplotlib, there's currently 2.0.0, 2.0.1, 2.2.0, 2.2.2 and 7 unpinned.
I'm proposing to make this 2.0.2 (new min), 2.1.0, 2.2.0, 2.2.* (LTS!), 3.0.0, 3.0.1, 3.0.* and 4 unpinned.
|
I like @h-vetinari's proposed pins in |
|
@jreback @TomAugspurger |
There was a problem hiding this comment.
LGTM.
@h-vetinari if you're interested, I believe that this will let us delete the following on SparseArray / SparseSeries
__array_wrap____array_prepare____array_priority__
Those are disabled when the class defines __array_ufunc__, which was started in NumPy 1.13
|
@TomAugspurger @jreback |
I've got an initial draft prepared, but will wait until this one is merged. |
|
this needs to wait until 0.24.2 is done and needs review |
ci/deps/travis-36.yaml
Outdated
| @@ -11,10 +11,10 @@ dependencies: | |||
| - gcsfs | |||
| - geopandas | |||
| - html5lib | |||
| - matplotlib | |||
| - matplotlib=3.0.0 | |||
There was a problem hiding this comment.
is there a reason this is pinned exactly to 3.0.0, which was superseded immedately by 3.0.1?
There was a problem hiding this comment.
There were almost three months between 3.0.0 and 3.0.1, see https://github.com/matplotlib/matplotlib/releases.
I just tried to spread some pins throughout the 3.x Series, see OP.
pandas/tests/plotting/common.py
Outdated
| @@ -53,8 +53,10 @@ def setup_method(self, method): | |||
| import matplotlib as mpl | |||
| mpl.rcdefaults() | |||
|
|
|||
| self.mpl_ge_2_0_1 = plotting._compat._mpl_ge_2_0_1() | |||
| self.mpl_eq_2_0_2 = plotting._compat._mpl_eq_2_0_2() | |||
There was a problem hiding this comment.
I would just bump mpl to 2.1 min to avoid all of these extra things
There was a problem hiding this comment.
ok, will do, but the special casing for 2.2.0 will still be necessary, unless we bump to 2.2.2
There was a problem hiding this comment.
cc @tacaswell any thoughts here?
There was a problem hiding this comment.
@jreback For pins, please check OP. Happy to change however you want. Will bump matplotlib pending further input from you (already to 2.2.2?)
ci/deps/travis-36.yaml
Outdated
| @@ -11,10 +11,10 @@ dependencies: | |||
| - gcsfs | |||
| - geopandas | |||
| - html5lib | |||
| - matplotlib | |||
| - matplotlib=3.0.0 | |||
There was a problem hiding this comment.
There were almost three months between 3.0.0 and 3.0.1, see https://github.com/matplotlib/matplotlib/releases.
I just tried to spread some pins throughout the 3.x Series, see OP.
pandas/tests/plotting/common.py
Outdated
| @@ -53,8 +53,10 @@ def setup_method(self, method): | |||
| import matplotlib as mpl | |||
| mpl.rcdefaults() | |||
|
|
|||
| self.mpl_ge_2_0_1 = plotting._compat._mpl_ge_2_0_1() | |||
| self.mpl_eq_2_0_2 = plotting._compat._mpl_eq_2_0_2() | |||
There was a problem hiding this comment.
ok, will do, but the special casing for 2.2.0 will still be necessary, unless we bump to 2.2.2
|
@jreback |
|
@jreback |
|
Sorry for the Matplotlib API thrashing. 2.2.x is the LTS version so it is a good choice! |
|
@jreback Bumping cython didn't help (resp. then the 3.5 jobs can't satisfy min). How about we drop the PY2 CI (#24942) first before bumping numpy? |
|
@h-vetinari can you merge master on this now that Py2 CI jobs are out? |
Since #24942 was split up, this PR here still depends on #25752. It would be helpful if you could give the latter one a look. :) |
|
Currently rebasing on top of #25752. This is the current set of pins for python/numpy/matplotlib. Comments welcome.
PS. Note that matplotlib |
|
see if we can remove 1 or more jobs keeping the same coverage (reasonably) of deps (can be a follow up) |
|
Well, I didn't design the CI jobs - there's OSes, numpies, locales, DB tests. Not sure we should reduce much more, but from the table above, I'd get rid of Maybe @TomAugspurger @jorisvandenbossche want to opine? Also, we still need to actually merge #25752. |
doc/source/whatsnew/v0.25.0.rst
Outdated
| Below is an overview of the minimal versions, as required resp. recommended. Optional libraries below the lowest tested version may still work, but are not considered supported. | ||
|
|
||
| +-----------------+-----------------+-----------------------+ | ||
| | Package | Minimum Version | Comment | |
There was a problem hiding this comment.
this is too verbose. pls separate into 2 tables, the basic deps (numpy, pytz, bottleneck, numexpr); make the table like we had it exactly
you can make a table for other things, but i am going to limit that list.
There was a problem hiding this comment.
I kept pytest due to #23519.
I kept the structure of the table because the distinction between the kind of requirement (hard requirement vs. requirement if installed) is clearer like this, and fits with the other table.
doc/source/whatsnew/v0.25.0.rst
Outdated
| +-----------------+-----------------+-----------------------+ | ||
| | numexpr | 2.6.2 | Required if installed | | ||
| +-----------------+-----------------+-----------------------+ | ||
| | beautifulsoup4 | 4.4.1 | Lowest tested version | |
There was a problem hiding this comment.
remove beautifulsoup, blosc, gcfs, junja2, lxml, psycopg2, pymsql, s3fs, cython and pytest
There was a problem hiding this comment.
pytest was explicitly added to this table by #23519.
I added all the libraries from https://pandas-docs.github.io/pandas-docs-travis/install.html#dependencies. I think this is a worthwhile overview to have somewhere (maybe not the whatsnew). Thoughts, @datapythonista?
doc/source/whatsnew/v0.25.0.rst
Outdated
| +-----------------+-----------------+-----------------------+ | ||
| | numexpr | 2.6.2 | Required if installed | | ||
| +-----------------+-----------------+-----------------------+ | ||
| | beautifulsoup4 | 4.4.1 | Lowest tested version | |
There was a problem hiding this comment.
pytest was explicitly added to this table by #23519.
I added all the libraries from https://pandas-docs.github.io/pandas-docs-travis/install.html#dependencies. I think this is a worthwhile overview to have somewhere (maybe not the whatsnew). Thoughts, @datapythonista?
doc/source/whatsnew/v0.25.0.rst
Outdated
| Below is an overview of the minimal versions, as required resp. recommended. Optional libraries below the lowest tested version may still work, but are not considered supported. | ||
|
|
||
| +-----------------+-----------------+-----------------------+ | ||
| | Package | Minimum Version | Comment | |
There was a problem hiding this comment.
I kept pytest due to #23519.
I kept the structure of the table because the distinction between the kind of requirement (hard requirement vs. requirement if installed) is clearer like this, and fits with the other table.
pandas/tests/test_sorting.py
Outdated
| r"')|" | ||
| r"unorderable types: int\(\) > datetime\.datetime\(\)") | ||
| msg = ("unorderable types: ({0} [<>] {1}|{1} [<>] {0})".format( | ||
| r"int\(\)", r"datetime\.datetime\(\)") # noqa: E126 |
There was a problem hiding this comment.
this fails the wheel building CI. pls don't revert this, I don't want to debug this problem. Just make this MUCH simpler so it will pretty much always pass.
There was a problem hiding this comment.
It will not fail the wheel building CI. I actually checked the wheel builder failures from #25867 before giving my review in #25874.
In case you don't have time to read the review, the original code in #25752 has an if _np_version_under1p14 else-switch that turned out to be wrong, while the regex here only uses a |-conjunction.
[...] I don't want to debug this problem.
Neither did I, but I've had to for a passing CI. Please have a look at the actual regexes. What I've written is a proper superset of the current regexes (again, now that the switch has been turned into "|").
There was a problem hiding this comment.
Just make this MUCH simpler so it will pretty much always pass.
Done.
doc/source/whatsnew/v0.25.0.rst
Outdated
| +-----------------+-----------------+-----------------------+ | ||
| | numexpr | 2.6.2 | Required if installed | | ||
| +-----------------+-----------------+-----------------------+ | ||
| | pytest | 4.0.2 | Development-only req. | |
There was a problem hiding this comment.
this is a misleading comment for pytest. its required if you run tests
Require if installed is very misleading, remove it
There was a problem hiding this comment.
I reverted it. That being said, pytest is not necessary to run pandas, so it is "as required" as, say openpyxl to write excel files.
doc/source/whatsnew/v0.25.0.rst
Outdated
| The following table lists the lowest version per library that is currently being tested throughout the development of pandas. | ||
| Optional libraries below the lowest tested version may still work, but are not considered supported. | ||
|
|
||
| +-----------------+-----------------+-----------------------+ |
There was a problem hiding this comment.
remove the comment column.
doc/source/whatsnew/v0.25.0.rst
Outdated
| +-----------------+-----------------+-----------------------+ | ||
| | numexpr | 2.6.2 | Required if installed | | ||
| +-----------------+-----------------+-----------------------+ | ||
| | pytest | 4.0.2 | Development-only req. | |
There was a problem hiding this comment.
I reverted it. That being said, pytest is not necessary to run pandas, so it is "as required" as, say openpyxl to write excel files.
doc/source/whatsnew/v0.25.0.rst
Outdated
| The following table lists the lowest version per library that is currently being tested throughout the development of pandas. | ||
| Optional libraries below the lowest tested version may still work, but are not considered supported. | ||
|
|
||
| +-----------------+-----------------+-----------------------+ |
pandas/tests/test_sorting.py
Outdated
| r"')|" | ||
| r"unorderable types: int\(\) > datetime\.datetime\(\)") | ||
| msg = ("unorderable types: ({0} [<>] {1}|{1} [<>] {0})".format( | ||
| r"int\(\)", r"datetime\.datetime\(\)") # noqa: E126 |
There was a problem hiding this comment.
It will not fail the wheel building CI. I actually checked the wheel builder failures from #25867 before giving my review in #25874.
In case you don't have time to read the review, the original code in #25752 has an if _np_version_under1p14 else-switch that turned out to be wrong, while the regex here only uses a |-conjunction.
[...] I don't want to debug this problem.
Neither did I, but I've had to for a passing CI. Please have a look at the actual regexes. What I've written is a proper superset of the current regexes (again, now that the switch has been turned into "|").
|
thanks |
|
@h-vetinari looks like this broke the gbq tests; these are pretty funky and I don't think are tested on actual branches (thoguh they used to be AFAIR): see https://travis-ci.org/pandas-dev/pandas/jobs/512494733 so pls do a PR to disable for now. |
I did fail, and I had fixed it (not by reverting exactly, but then I've tried to explain that already twice). Now it's more simplified anyway, which is IMO anyway a better approach for matching these error messages. |
#25227 is trying to add
pyproject.tomlwhich is necessary for the wholepip-machinery to work smoothly post v.19.0. However, this seems to be running into problems with a numpy version bump, and mood there was to bump numpy requirement to 1.13.x.The bare minimum for the bump is implemented in that PR, but a version bump requires getting rid of a bunch of compat code (and CI adjustments), otherwise the cruft just accumulates (it took me three PRs to clean out until 1.12).
Scipy is goint to bump to 1.13.3 with the next release (due to cython issues, I believe), so I thought I'll directly take it to 1.13.3 here.
I tried redistributing the CI jobs as best as I could. Here's the before/after:
EDIT: updated pins in #25554 (comment)
ci/deps/azure-27-compat.yaml:10:numpy=1.12.0numpy=1.13.3ci/deps/azure-27-locale.yaml:10:numpy=1.12.0numpy=1.14.*ci/deps/azure-36-locale_slow.yaml:16:numpynumpy=1.15.*ci/deps/azure-37-locale.yaml:15:numpynumpyci/deps/azure-37-numpydev.yaml:18:numpynumpyci/deps/azure-macos-35.yaml:14:numpy=1.12.0numpy=1.13.3ci/deps/azure-windows-27.yaml:15:numpy=1.12*numpy=1.13.3ci/deps/azure-windows-36.yaml:12:numpy=1.14*numpy=1.15.*ci/deps/travis-27.yaml:20:numpy=1.13*numpy=1.14.*ci/deps/travis-36-doc.yaml:23:numpy=1.13*numpyci/deps/travis-36-locale.yaml:15:numpynumpyci/deps/travis-36-slow.yaml:12:numpynumpyci/deps/travis-36.yaml:17:numpynumpy=1.15.*ci/deps/travis-37.yaml:10:numpynumpyMatplotlib needs to be bumped as well (since current min 2.0.0 cannot be resolved by conda together with numpy 1.13.3), but it's only a tiny bump to
2.0.22.1.02.1.12.2.2. In any case, the version spread should probably start to reflect that there's a bunch of mpl 3.0.x about, and so I also changed the spread a bit here:ci/deps/azure-27-locale.yaml:9:matplotlib=2.0.0matplotlib=2.2.2ci/deps/azure-36-locale_slow.yaml:13:matplotlibmatplotlib=3.0.*ci/deps/azure-37-locale.yaml:12:matplotlibmatplotlibci/deps/azure-macos-35.yaml:11:matplotlib=2.2.0matplotlib=2.2.3ci/deps/azure-windows-27.yaml:13:matplotlib=2.0.1matplotlib=2.2.4ci/deps/azure-windows-36.yaml:10:matplotlibmatplotlibci/deps/travis-27.yaml:16:matplotlib=2.2.2matplotlib=2.2.*ci/deps/travis-36-doc.yaml:17:matplotlibmatplotlibci/deps/travis-36-locale.yaml:12:matplotlibmatplotlib=3.0.1ci/deps/travis-36-slow.yaml:10:matplotlibmatplotlibci/deps/travis-36.yaml:14:matplotlibmatplotlib=3.0.0It's probably wishful thinking that everything will pass straight away (especially when mpl versions change, see the code in
pandas/tests/plotting/test_datetimelike.py), but oh well...