MAINT: Backports for 1.4.0rc2 #11161

tylerjereddy · 2019-12-02T02:16:52Z

Contains:

a commit setting version to 1.4.0rc2 unreleased
Backports:
release notes adjustments for the above backports

Currently Blocked By:

review/approval/merge of:
- ~~BUG: Fix random state bug multiscale_graphcorr #11152~~
- ~~BUG: empty sparse slice shapes #11166~~

* set version to 1.4.0rc2 unreleased

…ctors

…ortional weights

It is causing some errors in documentation build due to unused references

change a few size/index related variables from int -> int_p disable wraparound and boundscheck in a few other places

This is the non-default code path, not realised usually.

The issue could occur for CSR matrices with row start index > stop index. This fixes the same issue for column slice of CSC matrices.

* add out-of-bounds and invalid sparse indexing test cases requested by reviewer * adjust compressed.py source code such that start index > stop index case now returns an empty object (of same class as input)

This reverts commit 1e870bb.

* account for backported PRs: 11081, 11096, 11124, 11126, 11127

* BUG: Fix random state bug multiscale_graphcorr * BUG: Fix index error in multiscale_graphcorr * BUG: fix nonetype exception * ENH: Remove workers dependence for random_state in multiscale_graphcorr * MAIN: Move random state keyword to end of parameter list * MAIN: change list comprehension so it is more clear * MAIN: fix typo in seed parameter * ENH: Change seed generation in multiscale_graphcorr * DOC: Use random_state keyword in example * MAIN: Add whitespace around bitwise operator

* BUG: empty sparse slice shapes * CSC and CSR sparse matrix slices that return empty matrices should now correctly preserve a shape of (0, original_shape[1]) * refactor _process_slice replace use of 'empty' with i0=i1 which gives an empty slice this will keep the existing logic for shape handling * TST: add additional CSR and CSC slicing test cases

tylerjereddy · 2019-12-04T04:33:30Z

Updated with more backports on Dec. 3/2019 & corresponding release notes adjustments. The list of backports in the original description has been updated for a quick glance.

If CI is all green after this I'll be pretty tempted to merge & move forward with 1.4.0rc2.

tylerjereddy · 2019-12-04T04:41:07Z

doc/release/1.4.0-notes.rst

@@ -392,6 +391,7 @@ Authors
 =======

 * @endolith
+* @wenhui-prudencemed +


@wenhui-prudencemed I've added your GitHub handle to the updated release notes---apologies in advance if you have another preference for naming & we end up merging first (fairly likely considering recent delays!).

tylerjereddy · 2019-12-04T18:01:41Z

Azure failure is Python 3.8/ Windows:

================================== FAILURES ===================================
_______________ sparse/linalg/eigen/lobpcg/tests/test_lobpcg.py _______________
[gw1] win32 -- Python 3.8.0 C:\hostedtoolcache\windows\Python\3.8.0\x64\python.exe
worker 'gw1' crashed while running 'sparse/linalg/eigen/lobpcg/tests/test_lobpcg.py::test_hermitian'

Maybe that's a fluke (@lobpcg ?).

Travis failure is related to matrix entries pulling NumPy pre-release with the change I never really liked:

DeprecationWarning: Creating an ndarray with automatic object dtype is deprecated, use dtype=object if you intended it, otherwise specify an exact dtype

@rgommers @mattip @charris Should I just pin the NumPy versions for the SciPy 1.4.x maintenance branch to avoid NumPy pre-release? But then I guess it may be preferable to empower forward-compatibility and future backports, etc..

rgommers · 2019-12-04T18:19:19Z

The ragged array related deprecation warnings seem to be a mix between things that should be cleaned up in our code (e.g. the obrientransform failure looks unexpected) and things that look hard to understand.

We'll discuss it in an hour in the triage call, for now just ignore the failures I'd say @tylerjereddy. No point messing with CI pinning, either we'll clean it all up asap or we'll revert the change in NumPy for a while.

rgommers · 2019-12-04T18:23:12Z

The lobpcg crash needs looking into. Should be fine to ignore for the release @tylerjereddy unless it happens again.

These are the relevant lines:

  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\scipy\linalg\decomp.py", line 455 in eigh
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\scipy\sparse\linalg\eigen\lobpcg\lobpcg.py", line 580 in lobpcg

Full traceback:

......................................................................... [ 62%]
Windows fatal exception: access violation

Thread 0x00000b70 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\execnet\gateway_base.py", line 400 in read
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\execnet\gateway_base.py", line 432 in from_io
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\execnet\gateway_base.py", line 967 in _thread_receiver
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\execnet\gateway_base.py", line 220 in run
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\execnet\gateway_base.py", line 285 in _perform_spawn

Current thread 0x00001138 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\scipy\linalg\decomp.py", line 455 in eigh
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\scipy\sparse\linalg\eigen\lobpcg\lobpcg.py", line 580 in lobpcg
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\scipy\sparse\linalg\eigen\lobpcg\tests\test_lobpcg.py", line 231 in test_hermitian
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\_pytest\python.py", line 166 in pytest_pyfunc_call
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\pluggy\manager.py", line 84 in <lambda>
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\pluggy\manager.py", line 93 in _hookexec
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\_pytest\python.py", line 1435 in runtest
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\_pytest\runner.py", line 131 in pytest_runtest_call
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\pluggy\manager.py", line 84 in <lambda>
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\pluggy\manager.py", line 93 in _hookexec
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\_pytest\runner.py", line 207 in <lambda>
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\_pytest\runner.py", line 234 in from_call
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\_pytest\runner.py", line 206 in call_runtest_hook
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\_pytest\runner.py", line 182 in call_and_report
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\_pytest\runner.py", line 96 in runtestprotocol
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\_pytest\runner.py", line 81 in pytest_runtest_protocol
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\pluggy\manager.py", line 84 in <lambda>
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\pluggy\manager.py", line 93 in _hookexec
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\xdist\remote.py", line 87 in run_one_test
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\xdist\remote.py", line 70 in pytest_runtestloop
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\pluggy\manager.py", line 84 in <lambda>
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\pluggy\manager.py", line 93 in _hookexec
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\_pytest\main.py", line 246 in _main
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\_pytest\main.py", line 196 in wrap_session
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\_pytest\main.py", line 239 in pytest_cmdline_main
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\pluggy\manager.py", line 84 in <lambda>
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\pluggy\manager.py", line 93 in _hookexec
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\xdist\remote.py", line 258 in <module>
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\execnet\gateway_base.py", line 1084 in executetask
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\execnet\gateway_base.py", line 220 in run
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\execnet\gateway_base.py", line 285 in _perform_spawn
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\execnet\gateway_base.py", line 267 in integrate_as_primary_thread
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\execnet\gateway_base.py", line 1060 in serve
  File "C:\hostedtoolcache\windows\Python\3.8.0\x64\lib\site-packages\execnet\gateway_base.py", line 1554 in serve
  File "<string>", line 8 in <module>
  File "<string>", line 1 in <module>
........................................................................ [ 62%]
........................................................................ [ 62%]
........................................................................ [ 62%]
..................................[gw1] node down: Not properly terminated
f
replacing crashed worker gw1
........................................................................ [ 63%]

EDIT: that's a crash in eigh in a call to a LAPACK function (one of the *gvd ones), so likely an OpenBLAS issue.

lobpcg · 2019-12-04T18:32:43Z

It is possible for LOBPCG to crash, in principle, although that is very rare. I do not recollect seeing it crashing in this test. Please try to re-run just with a different seed in https://github.com/scipy/scipy/blob/master/scipy/sparse/linalg/eigen/lobpcg/tests/test_lobpcg.py line 209 np.random.seed(1234) - just use any other number instead of 1234. If it crashes again, an investigation would be needed.

tylerjereddy · 2019-12-05T04:26:11Z

I ran the test 1000 times with the same/original seed using pytest-repeat, a site.cfg pointing to OpenBLAS 0.3.7 on Linux/Python 3.7 with:

python runtests.py -t "scipy/sparse/linalg/eigen/lobpcg/tests/test_lobpcg.py::test_hermitian" -- --count=1000 -x -n 2.

It was always successful: 1000 passed in 99.01s (0:01:39).

I guess I'll try with Python 3.8 now. Doing it on Windows may take a bit more work for me locally.

tylerjereddy · 2019-12-05T04:55:43Z

Stable after 2500 repeats on Python 3.8/Linux. For Windows maybe I will check on my Azure fork tomorrow by replacing the normal test suite with just a large repeat count of the test in question, if we want to be safe.

tylerjereddy · 2019-12-05T05:00:48Z

I did those tests on latest master though..

lobpcg · 2019-12-05T05:01:22Z

Unless the hardware is faulty, the repeats must be absolutely identical, with all the digits, i.e. no reason for repeats.

tylerjereddy · 2019-12-05T17:08:35Z

Alright, close/reopen to flush CI again. Hopefully the lobpcg issue just disappears.

If I'm not mistaken we may need to be patient for the object=dtype matter---as far as I can tell that reversion has so far been restricted to the 1.18 branch of NumPy and not the nightly wheels we pull in for two Travis entries.

WarrenWeckesser · 2019-12-05T17:27:41Z

I'm not sure if this is the right PR in which to report this, but I just noticed that the addition of scipy.stats.multiscale_graphcorr is listed twice in the release notes in the "scipy.stats improvements" section.

Also in the release notes, a blank line is needed before

The generalized inverse Gaussian distribution (`scipy.stats.geninvgauss`) has  been added.

to separate it from the previous item.

tylerjereddy · 2019-12-05T20:45:37Z

lobpcg issue went away; new ppc64le issue is likely the usual sporadic/unrelated failure; just the deprecations left now it seems; I'll try to address the release notes matters later---those changes can cause CI to fail too if I'm not careful, but anyway..

* update 1.4.0 release notes to reflect backported PRs: 11152, 11166, 11167 * remove a redundant mention of multiscale_graphcorr in the release notes and improve spacing around the original mention of multiscale_graphcorr

tylerjereddy · 2019-12-07T14:11:50Z

I addressed Warren's release notes corrections and pushed those in---this should also serve to flush CI and tell us if nightly NumPy wheels will give us a green light now.

tylerjereddy · 2019-12-07T17:49:55Z

Ok, still have the dtype=object problem, for now.

charris · 2019-12-07T18:15:49Z

I just triggered a wheels build of master, the last was two days ago.

charris · 2019-12-07T21:19:24Z

Reran the failing tests, they are passing now.

tylerjereddy · 2019-12-07T21:24:45Z

Great, thanks Chuck, merging backport PR with CI all green.

tylerjereddy and others added 22 commits December 1, 2019 18:41

REL: set version to 1.4.0rc2

e08a742

* set version to 1.4.0rc2 unreleased

API: Restore Rotation.match_vectors, deprecated in favors of align_ve…

ed7c6ca

…ctors

ENH: Return loss value from Rotation.align_vectors

2578840

TST: Add a test of Rotation.align_vectors comparing results with prop…

ac0429b

…ortional weights

MAINT: Change error to warning in Rotation.align_vectors

706f8db

MAINT: Add lower bound of 0 for returned loss in Rotation.align_vectors

78debee

API: Return RMSD instead of loss function from Rotation.align_vectors

d521c00

MAINT: Remove docstring of deprecated match_vectors

dd3603c

It is causing some errors in documentation build due to unused references

API: Add return_sensitivity argument to Rotation.align_vectors

a2b1c5e

MAINT: Make Rotation.align_vectors work with single observations

67051f3

TST: Add tests with invalid inputs to Rotation.align_vectors

42e2501

BUG: remove repeated insertion of the same node to the heap in dijkstra

5376c2c

MAINT: move current_node=&nodes[j_source] line up

5e506a2

add missing types to _stats.pyx

0ab95d4

change a few size/index related variables from int -> int_p disable wraparound and boundscheck in a few other places

avoid warning about a potentially uninitialized variable

93476e6

BUG: interpolate/fitpack: fix leak in splprep for task!=0

d5ec41c

This is the non-default code path, not realised usually.

BUG: interpolate/fitpack: fix memory leak in splprep

887b8f9

FIX: Avoid potential segfault when slicing CSR or CSC matrices

11d1a1d

The issue could occur for CSR matrices with row start index > stop index. This fixes the same issue for column slice of CSC matrices.

TST: add test cases for invalid indexing of CSR/CSC matrices

531eda9

MAINT: PR 11127 reviewer adjustments

23f1fc0

* add out-of-bounds and invalid sparse indexing test cases requested by reviewer * adjust compressed.py source code such that start index > stop index case now returns an empty object (of same class as input)

Revert "TST: add test cases for invalid indexing of CSR/CSC matrices"

d5fe396

This reverts commit 1e870bb.

DOC: update 1.4.0 release notes

dee48db

* account for backported PRs: 11081, 11096, 11124, 11126, 11127

tylerjereddy added maintenance Items related to regular maintenance tasks needs-work Items that are pending response from the author labels Dec 2, 2019

sampan501 and others added 3 commits December 3, 2019 21:17

remove redundant fft in signal.resample

b61c951

tylerjereddy changed the title ~~WIP, MAINT: Backports for 1.4.0rc2~~ MAINT: Backports for 1.4.0rc2 Dec 4, 2019

tylerjereddy removed the needs-work Items that are pending response from the author label Dec 4, 2019

tylerjereddy commented Dec 4, 2019

View reviewed changes

This was referenced Dec 4, 2019

"Controlled" creation of object arrays numpy/numpy#15047

Closed

MAINT: dealing with NumPy pre-release test failures #11172

Closed

tylerjereddy closed this Dec 5, 2019

tylerjereddy reopened this Dec 5, 2019

grlee77 mentioned this pull request Dec 6, 2019

update slicing of CSR and CSC matrices for compatibility with SciPy 1.4.0 cupy/cupy#2776

Merged

DOC: update 1.4.0 release notes

7bd5ad6

* update 1.4.0 release notes to reflect backported PRs: 11152, 11166, 11167 * remove a redundant mention of multiscale_graphcorr in the release notes and improve spacing around the original mention of multiscale_graphcorr

tylerjereddy force-pushed the backports_140_rc2 branch from b8c1715 to 7bd5ad6 Compare December 7, 2019 14:10

tylerjereddy merged commit 6a00a39 into scipy:maintenance/1.4.x Dec 7, 2019

tylerjereddy deleted the backports_140_rc2 branch December 7, 2019 21:25

tylerjereddy added this to the 1.4.0 milestone Dec 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MAINT: Backports for 1.4.0rc2 #11161

MAINT: Backports for 1.4.0rc2 #11161

tylerjereddy commented Dec 2, 2019 •

edited

tylerjereddy commented Dec 4, 2019

tylerjereddy Dec 4, 2019

tylerjereddy commented Dec 4, 2019

rgommers commented Dec 4, 2019

rgommers commented Dec 4, 2019 •

edited

lobpcg commented Dec 4, 2019

tylerjereddy commented Dec 5, 2019

tylerjereddy commented Dec 5, 2019

tylerjereddy commented Dec 5, 2019

lobpcg commented Dec 5, 2019

tylerjereddy commented Dec 5, 2019 •

edited

WarrenWeckesser commented Dec 5, 2019

tylerjereddy commented Dec 5, 2019

tylerjereddy commented Dec 7, 2019

tylerjereddy commented Dec 7, 2019

charris commented Dec 7, 2019

charris commented Dec 7, 2019

tylerjereddy commented Dec 7, 2019

MAINT: Backports for 1.4.0rc2 #11161

MAINT: Backports for 1.4.0rc2 #11161

Conversation

tylerjereddy commented Dec 2, 2019 • edited

tylerjereddy commented Dec 4, 2019

tylerjereddy Dec 4, 2019

Choose a reason for hiding this comment

tylerjereddy commented Dec 4, 2019

rgommers commented Dec 4, 2019

rgommers commented Dec 4, 2019 • edited

lobpcg commented Dec 4, 2019

tylerjereddy commented Dec 5, 2019

tylerjereddy commented Dec 5, 2019

tylerjereddy commented Dec 5, 2019

lobpcg commented Dec 5, 2019

tylerjereddy commented Dec 5, 2019 • edited

WarrenWeckesser commented Dec 5, 2019

tylerjereddy commented Dec 5, 2019

tylerjereddy commented Dec 7, 2019

tylerjereddy commented Dec 7, 2019

charris commented Dec 7, 2019

charris commented Dec 7, 2019

tylerjereddy commented Dec 7, 2019

tylerjereddy commented Dec 2, 2019 •

edited

rgommers commented Dec 4, 2019 •

edited

tylerjereddy commented Dec 5, 2019 •

edited