DOC: advise against use of matrix. #10973
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rgommers
approved these changes
Apr 25, 2018
2 tasks
|
Thanks Marten. |
johnyf
added a commit
to tulip-control/tulip-control
that referenced
this pull request
Jun 10, 2021
because matrices have been deprecated in `numpy`. Usage of `scipy.sparse` is causing this issue when conversions to matrices are performed. I changed the: - calls to the method `scipy.sparse.lil.lil_matrix.todense` - to calls to the method `scipy.sparse.lil.lil_matrix.toarray`, to avoid the conversions that raise `PendingDeprecationWarning`s. The warnings are raised by the call to the method `lil_matrix.todense` because this call involves the instantiation of the class `numpy.matrix`, which is deprecated. In contrast, the method `lil_matrix.toarray` creates an instance of `numpy.ndarray`. (In fact, in the module `tulip.abstract.discretization`, wherever the method `lil_matrix.todense` was called, the value that it returned was immediately converted to a `numpy.ndarray`. So calling the method `lil_matrix.toarray` is actually more efficient.) The class `numpy.matrix` is deprecated and will probably be removed in the future. This will happen after arrangements have been made for `scipy.space`. (For these points and more information, read the references listed at the end.) Still, I do not think that continuing to use `scipy.sparse.lil_matrix` until when `numpy` removes matrices is a safe approach. Instead, using `numpy.ndarray` would be safer. Moreover, I do think that there are other data structures that would fit this use case better than sparse matrices. ## Diagnosis I describe below the approach I (eventually) followed to debug this warning, because finding the cause was difficult. The issue is a `PendingDeprecationWarning` issued from `numpy`. This warning is visible in `pytest` runs, but *not* when running the Python test file directly. Moreover, `pytest` reports the warning, and from which test function it originates. The warning itself reads (I have wrapped the lines here): ``` ===================================== warnings summary ====================================== abstract_test.py::transition_directions_test /.../.virtualenvs/.../lib/python3.9/site-packages/numpy/matrixlib/defmatrix.py:69: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. return matrix(data, dtype=dtype, copy=False) ``` So the line in `tulip` that causes the warning cannot be found from the information contained in the warning. The above `PendingDeprecationWarning` was introduced in `numpy` in commit: numpy/numpy@11e9d2a This warning was then ignored in the module `scipy.sparse.__init__`, in `scipy` commit: scipy/scipy@a874bd5 It appears that this configuration of warnings by `scipy` interacts with `pytest` complexly: - when running with `pytest abstract_test.py`, the `PendingDeprecationWarning` is visible, but - when running with `python -X dev -- abstract_test.py`, the `PendingDeprecationWarning` is invisible. This behavior is due to the call: ```python warnings.filterwarnings( 'ignore', message='the matrix subclass is not the recommended way') ``` within `scipy.sparse.__init__.py` (introduced in the `scipy` commit that was mentioned above). Read also: https://docs.python.org/3/library/exceptions.html#PendingDeprecationWarning https://www.python.org/dev/peps/pep-0565/ As a result, it is difficult to find the cause within `tulip` of this `PendingDeprecationWarning`. ## Getting a traceback The test function that triggered the `PendingDeprecationWarning` from `numpy` was not failing, so there was no traceback that would indicate which line in `tulip` caused the warning. In addition, there was an earlier warning issued by `matplotlib`. So turning warnings to errors with the argument `-Werror` would cause `pytest` to turn the `matplotlib` warning into an error, and stop before the warning of interest: ```shell pytest -Werror abstract_test.py ``` So first I removed the `matplotlib` warnings (temporarily), by commenting the line `matplotlib.use('Agg')` in the file `abstract_test.py`. This made the warning of interest to become the first warning. I then passed `-Werror` to `pytest`, and this turned the `numpy` warning into an error, which produced the traceback shown below: (The cause of these `matplotlib` warnings (there are two) is in the package `polytope`, and has been addressed there, in commit: tulip-control/polytope@c464818 These changes will become available to `tulip` with the next `polytope` release. Until then, the CI tests of `tulip` will raise these `matplotlib` warnings. These warnings could be explicitly ignored by using `with pytest.warns`.) (The paths to `tulip` in the traceback below lead to the repository's `tulip`, instead of a directory under Python's `site-packages`, because during this phase of debugging I installed `tulip` with `pip install -e .`, to iterate faster while debugging.) ``` ../tulip/abstract/discretization.py:1666: in discretize_switched plot_mode_partitions(merged_abstr, show_ts, only_adjacent) ../tulip/abstract/discretization.py:1673: in plot_mode_partitions axs = swab.plot(show_ts, only_adjacent) ../tulip/abstract/discretization.py:187: in plot ax = ab.plot(show_ts, only_adjacent, color_seed) ../tulip/abstract/discretization.py:403: in plot ax = _plot_abstraction(self, show_ts, only_adjacent, ../tulip/abstract/discretization.py:446: in _plot_abstraction ax = ab.ppp.plot( ../tulip/abstract/prop2partition.py:600: in plot return plot_partition( .../.virtualenvs/.../lib/python3.9/site-packages/polytope/plot.py:90: in plot_partition trans = nx.to_numpy_matrix(trans, nodelist=ppp2trans) .../.virtualenvs/.../lib/python3.9/site-packages/networkx/convert_matrix.py:553: in to_numpy_matrix M = np.asmatrix(A, dtype=dtype) .../.virtualenvs/.../lib/python3.9/site-packages/numpy/matrixlib/defmatrix.py:69: in asmatrix return matrix(data, dtype=dtype, copy=False) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ subtype = <class 'numpy.matrix'> data = array([[1., 0., 0., 0., 0., 0.], [0., 1., 1., 0., 1., 1.], [1., 0., 1., 1., 0., 1.], [1., 0., 0., 1., 0., 0.], [0., 0., 0., 0., 1., 1.], [1., 0., 0., 0., 0., 1.]]) dtype = None, copy = False def __new__(subtype, data, dtype=None, copy=True): > warnings.warn('the matrix subclass is not the recommended way to ' 'represent matrices or deal with linear algebra (see ' 'https://docs.scipy.org/doc/numpy/user/' 'numpy-for-matlab-users.html). ' 'Please adjust your code to use regular ndarray.', PendingDeprecationWarning, stacklevel=2) E PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. .../.virtualenvs/.../lib/python3.9/site-packages/numpy/matrixlib/defmatrix.py:116: PendingDeprecationWarning ``` As the traceback shows, the issue is due to a call to the function `networkx.to_numpy_matrix` within the function `polytope.plot.plot_partition`. So avoiding this warning will be possible after the next release of the package `polytope`. (Note that inserting an `assert False` in a suitable line within the function `transition_directions_test` is not an alternative to passing the argument `-Werror`, because the `assert False` will result in a traceback where the `assert` statement appears, instead of a traceback that shows the call stack at the point where the warning was issued.) ## Speeding up debugging using `pytest` Also, since I had to be running `pytest` on the Python file `abstract_test.py`, `pytest` would collect all test functions, and run them. The file `abstract_test.py` happens to contain several slow test functions, so running them all just to observe the results for the one function of interest is not time-efficient. What I did to speed up runs was to rename all `test_*` functions contained in `abstract_test.py`, except for the one function of interest (namely `transition_directions_test`), to identifiers outside the patterns collected by `pytest`. A simpler alternative, for use with larger test files, is to do the opposite: rename only the function of interest to a different pattern, and then change the line `python_functions = ` in the configuration file `pytest.ini`. ## References - numpy/numpy#10142 (DEP: Pending deprecation warning for matrix) - numpy/numpy#10973 (DOC: advise against use of matrix) - scipy/scipy#8887 (MAINT: filter out np.matrix PendingDeprecationWarning's in numpy >=1.15) - scipy/scipy#9734 (PendingDeprecationWarning for np.matrix with pytest) - scikit-learn/scikit-learn#12327 (PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices) - scikit-learn/scikit-learn#13076 ([MRG] Ignore PendingDepWarnings of matrix subclass with pytest) - cvxpy/cvxpy#567 (NumPy matrix class is pending deprecation and issuing warnings) - cvxpy/cvxpy#637 (RF: Use a 2D np array instead of matrix to represent scalars) - cvxpy/cvxpy#638 (RF: Change np.matrix to np.array in several places) - cvxpy/cvxpy#644 (PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra) - https://docs.pytest.org/en/6.2.x/warnings.html#deprecationwarning-and-pendingdeprecationwarning
johnyf
added a commit
to tulip-control/tulip-control
that referenced
this pull request
Jun 20, 2021
because matrices have been deprecated in `numpy`. Usage of `scipy.sparse` is causing this issue when conversions to matrices are performed. I changed the: - calls to the method `scipy.sparse.lil.lil_matrix.todense` - to calls to the method `scipy.sparse.lil.lil_matrix.toarray`, to avoid the conversions that raise `PendingDeprecationWarning`s. The warnings are raised by the call to the method `lil_matrix.todense` because this call involves the instantiation of the class `numpy.matrix`, which is deprecated. In contrast, the method `lil_matrix.toarray` creates an instance of `numpy.ndarray`. (In fact, in the module `tulip.abstract.discretization`, wherever the method `lil_matrix.todense` was called, the value that it returned was immediately converted to a `numpy.ndarray`. So calling the method `lil_matrix.toarray` is actually more efficient.) The class `numpy.matrix` is deprecated and will probably be removed in the future. This will happen after arrangements have been made for `scipy.space`. (For these points and more information, read the references listed at the end.) Still, I do not think that continuing to use `scipy.sparse.lil_matrix` until when `numpy` removes matrices is a safe approach. Instead, using `numpy.ndarray` would be safer. Moreover, I do think that there are other data structures that would fit this use case better than sparse matrices. ## Diagnosis I describe below the approach I (eventually) followed to debug this warning, because finding the cause was difficult. The issue is a `PendingDeprecationWarning` issued from `numpy`. This warning is visible in `pytest` runs, but *not* when running the Python test file directly. Moreover, `pytest` reports the warning, and from which test function it originates. The warning itself reads (I have wrapped the lines here): ``` ===================================== warnings summary ====================================== abstract_test.py::transition_directions_test /.../.virtualenvs/.../lib/python3.9/site-packages/numpy/matrixlib/defmatrix.py:69: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. return matrix(data, dtype=dtype, copy=False) ``` So the line in `tulip` that causes the warning cannot be found from the information contained in the warning. The above `PendingDeprecationWarning` was introduced in `numpy` in commit: numpy/numpy@11e9d2a This warning was then ignored in the module `scipy.sparse.__init__`, in `scipy` commit: scipy/scipy@a874bd5 It appears that this configuration of warnings by `scipy` interacts with `pytest` complexly: - when running with `pytest abstract_test.py`, the `PendingDeprecationWarning` is visible, but - when running with `python -X dev -- abstract_test.py`, the `PendingDeprecationWarning` is invisible. This behavior is due to the call: ```python warnings.filterwarnings( 'ignore', message='the matrix subclass is not the recommended way') ``` within `scipy.sparse.__init__.py` (introduced in the `scipy` commit that was mentioned above). Read also: https://docs.python.org/3/library/exceptions.html#PendingDeprecationWarning https://www.python.org/dev/peps/pep-0565/ As a result, it is difficult to find the cause within `tulip` of this `PendingDeprecationWarning`. ## Getting a traceback The test function that triggered the `PendingDeprecationWarning` from `numpy` was not failing, so there was no traceback that would indicate which line in `tulip` caused the warning. In addition, there was an earlier warning issued by `matplotlib`. So turning warnings to errors with the argument `-Werror` would cause `pytest` to turn the `matplotlib` warning into an error, and stop before the warning of interest: ```shell pytest -Werror abstract_test.py ``` So first I removed the `matplotlib` warnings (temporarily), by commenting the line `matplotlib.use('Agg')` in the file `abstract_test.py`. This made the warning of interest to become the first warning. I then passed `-Werror` to `pytest`, and this turned the `numpy` warning into an error, which produced the traceback shown below: (The cause of these `matplotlib` warnings (there are two) is in the package `polytope`, and has been addressed there, in commit: tulip-control/polytope@c464818 These changes will become available to `tulip` with the next `polytope` release. Until then, the CI tests of `tulip` will raise these `matplotlib` warnings. These warnings could be explicitly ignored by using `with pytest.warns`.) (The paths to `tulip` in the traceback below lead to the repository's `tulip`, instead of a directory under Python's `site-packages`, because during this phase of debugging I installed `tulip` with `pip install -e .`, to iterate faster while debugging.) ``` ../tulip/abstract/discretization.py:1666: in discretize_switched plot_mode_partitions(merged_abstr, show_ts, only_adjacent) ../tulip/abstract/discretization.py:1673: in plot_mode_partitions axs = swab.plot(show_ts, only_adjacent) ../tulip/abstract/discretization.py:187: in plot ax = ab.plot(show_ts, only_adjacent, color_seed) ../tulip/abstract/discretization.py:403: in plot ax = _plot_abstraction(self, show_ts, only_adjacent, ../tulip/abstract/discretization.py:446: in _plot_abstraction ax = ab.ppp.plot( ../tulip/abstract/prop2partition.py:600: in plot return plot_partition( .../.virtualenvs/.../lib/python3.9/site-packages/polytope/plot.py:90: in plot_partition trans = nx.to_numpy_matrix(trans, nodelist=ppp2trans) .../.virtualenvs/.../lib/python3.9/site-packages/networkx/convert_matrix.py:553: in to_numpy_matrix M = np.asmatrix(A, dtype=dtype) .../.virtualenvs/.../lib/python3.9/site-packages/numpy/matrixlib/defmatrix.py:69: in asmatrix return matrix(data, dtype=dtype, copy=False) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ subtype = <class 'numpy.matrix'> data = array([[1., 0., 0., 0., 0., 0.], [0., 1., 1., 0., 1., 1.], [1., 0., 1., 1., 0., 1.], [1., 0., 0., 1., 0., 0.], [0., 0., 0., 0., 1., 1.], [1., 0., 0., 0., 0., 1.]]) dtype = None, copy = False def __new__(subtype, data, dtype=None, copy=True): > warnings.warn('the matrix subclass is not the recommended way to ' 'represent matrices or deal with linear algebra (see ' 'https://docs.scipy.org/doc/numpy/user/' 'numpy-for-matlab-users.html). ' 'Please adjust your code to use regular ndarray.', PendingDeprecationWarning, stacklevel=2) E PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. .../.virtualenvs/.../lib/python3.9/site-packages/numpy/matrixlib/defmatrix.py:116: PendingDeprecationWarning ``` As the traceback shows, the issue is due to a call to the function `networkx.to_numpy_matrix` within the function `polytope.plot.plot_partition`. So avoiding this warning will be possible after the next release of the package `polytope`. (Note that inserting an `assert False` in a suitable line within the function `transition_directions_test` is not an alternative to passing the argument `-Werror`, because the `assert False` will result in a traceback where the `assert` statement appears, instead of a traceback that shows the call stack at the point where the warning was issued.) ## Speeding up debugging using `pytest` Also, since I had to be running `pytest` on the Python file `abstract_test.py`, `pytest` would collect all test functions, and run them. The file `abstract_test.py` happens to contain several slow test functions, so running them all just to observe the results for the one function of interest is not time-efficient. What I did to speed up runs was to rename all `test_*` functions contained in `abstract_test.py`, except for the one function of interest (namely `transition_directions_test`), to identifiers outside the patterns collected by `pytest`. A simpler alternative, for use with larger test files, is to do the opposite: rename only the function of interest to a different pattern, and then change the line `python_functions = ` in the configuration file `pytest.ini`. ## References - numpy/numpy#10142 (DEP: Pending deprecation warning for matrix) - numpy/numpy#10973 (DOC: advise against use of matrix) - scipy/scipy#8887 (MAINT: filter out np.matrix PendingDeprecationWarning's in numpy >=1.15) - scipy/scipy#9734 (PendingDeprecationWarning for np.matrix with pytest) - scikit-learn/scikit-learn#12327 (PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices) - scikit-learn/scikit-learn#13076 ([MRG] Ignore PendingDepWarnings of matrix subclass with pytest) - cvxpy/cvxpy#567 (NumPy matrix class is pending deprecation and issuing warnings) - cvxpy/cvxpy#637 (RF: Use a 2D np array instead of matrix to represent scalars) - cvxpy/cvxpy#638 (RF: Change np.matrix to np.array in several places) - cvxpy/cvxpy#644 (PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra) - https://docs.pytest.org/en/6.2.x/warnings.html#deprecationwarning-and-pendingdeprecationwarning
johnyf
added a commit
to tulip-control/tulip-control
that referenced
this pull request
Aug 12, 2021
because matrices have been deprecated in `numpy`. Usage of `scipy.sparse` is causing this issue when conversions to matrices are performed. I changed the: - calls to the method `scipy.sparse.lil.lil_matrix.todense` - to calls to the method `scipy.sparse.lil.lil_matrix.toarray`, to avoid the conversions that raise `PendingDeprecationWarning`s. The warnings are raised by the call to the method `lil_matrix.todense` because this call involves the instantiation of the class `numpy.matrix`, which is deprecated. In contrast, the method `lil_matrix.toarray` creates an instance of `numpy.ndarray`. (In fact, in the module `tulip.abstract.discretization`, wherever the method `lil_matrix.todense` was called, the value that it returned was immediately converted to a `numpy.ndarray`. So calling the method `lil_matrix.toarray` is actually more efficient.) The class `numpy.matrix` is deprecated and will probably be removed in the future. This will happen after arrangements have been made for `scipy.space`. (For these points and more information, read the references listed at the end.) Still, I do not think that continuing to use `scipy.sparse.lil_matrix` until when `numpy` removes matrices is a safe approach. Instead, using `numpy.ndarray` would be safer. Moreover, I do think that there are other data structures that would fit this use case better than sparse matrices. ## Diagnosis I describe below the approach I (eventually) followed to debug this warning, because finding the cause was difficult. The issue is a `PendingDeprecationWarning` issued from `numpy`. This warning is visible in `pytest` runs, but *not* when running the Python test file directly. Moreover, `pytest` reports the warning, and from which test function it originates. The warning itself reads (I have wrapped the lines here): ``` ===================================== warnings summary ====================================== abstract_test.py::transition_directions_test /.../.virtualenvs/.../lib/python3.9/site-packages/numpy/matrixlib/defmatrix.py:69: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. return matrix(data, dtype=dtype, copy=False) ``` So the line in `tulip` that causes the warning cannot be found from the information contained in the warning. The above `PendingDeprecationWarning` was introduced in `numpy` in commit: numpy/numpy@11e9d2a This warning was then ignored in the module `scipy.sparse.__init__`, in `scipy` commit: scipy/scipy@a874bd5 It appears that this configuration of warnings by `scipy` interacts with `pytest` complexly: - when running with `pytest abstract_test.py`, the `PendingDeprecationWarning` is visible, but - when running with `python -X dev -- abstract_test.py`, the `PendingDeprecationWarning` is invisible. This behavior is due to the call: ```python warnings.filterwarnings( 'ignore', message='the matrix subclass is not the recommended way') ``` within `scipy.sparse.__init__.py` (introduced in the `scipy` commit that was mentioned above). Read also: https://docs.python.org/3/library/exceptions.html#PendingDeprecationWarning https://www.python.org/dev/peps/pep-0565/ As a result, it is difficult to find the cause within `tulip` of this `PendingDeprecationWarning`. ## Getting a traceback The test function that triggered the `PendingDeprecationWarning` from `numpy` was not failing, so there was no traceback that would indicate which line in `tulip` caused the warning. In addition, there was an earlier warning issued by `matplotlib`. So turning warnings to errors with the argument `-Werror` would cause `pytest` to turn the `matplotlib` warning into an error, and stop before the warning of interest: ```shell pytest -Werror abstract_test.py ``` So first I removed the `matplotlib` warnings (temporarily), by commenting the line `matplotlib.use('Agg')` in the file `abstract_test.py`. This made the warning of interest to become the first warning. I then passed `-Werror` to `pytest`, and this turned the `numpy` warning into an error, which produced the traceback shown below: (The cause of these `matplotlib` warnings (there are two) is in the package `polytope`, and has been addressed there, in commit: tulip-control/polytope@c464818 These changes will become available to `tulip` with the next `polytope` release. Until then, the CI tests of `tulip` will raise these `matplotlib` warnings. These warnings could be explicitly ignored by using `with pytest.warns`.) (The paths to `tulip` in the traceback below lead to the repository's `tulip`, instead of a directory under Python's `site-packages`, because during this phase of debugging I installed `tulip` with `pip install -e .`, to iterate faster while debugging.) ``` ../tulip/abstract/discretization.py:1666: in discretize_switched plot_mode_partitions(merged_abstr, show_ts, only_adjacent) ../tulip/abstract/discretization.py:1673: in plot_mode_partitions axs = swab.plot(show_ts, only_adjacent) ../tulip/abstract/discretization.py:187: in plot ax = ab.plot(show_ts, only_adjacent, color_seed) ../tulip/abstract/discretization.py:403: in plot ax = _plot_abstraction(self, show_ts, only_adjacent, ../tulip/abstract/discretization.py:446: in _plot_abstraction ax = ab.ppp.plot( ../tulip/abstract/prop2partition.py:600: in plot return plot_partition( .../.virtualenvs/.../lib/python3.9/site-packages/polytope/plot.py:90: in plot_partition trans = nx.to_numpy_matrix(trans, nodelist=ppp2trans) .../.virtualenvs/.../lib/python3.9/site-packages/networkx/convert_matrix.py:553: in to_numpy_matrix M = np.asmatrix(A, dtype=dtype) .../.virtualenvs/.../lib/python3.9/site-packages/numpy/matrixlib/defmatrix.py:69: in asmatrix return matrix(data, dtype=dtype, copy=False) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ subtype = <class 'numpy.matrix'> data = array([[1., 0., 0., 0., 0., 0.], [0., 1., 1., 0., 1., 1.], [1., 0., 1., 1., 0., 1.], [1., 0., 0., 1., 0., 0.], [0., 0., 0., 0., 1., 1.], [1., 0., 0., 0., 0., 1.]]) dtype = None, copy = False def __new__(subtype, data, dtype=None, copy=True): > warnings.warn('the matrix subclass is not the recommended way to ' 'represent matrices or deal with linear algebra (see ' 'https://docs.scipy.org/doc/numpy/user/' 'numpy-for-matlab-users.html). ' 'Please adjust your code to use regular ndarray.', PendingDeprecationWarning, stacklevel=2) E PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. .../.virtualenvs/.../lib/python3.9/site-packages/numpy/matrixlib/defmatrix.py:116: PendingDeprecationWarning ``` As the traceback shows, the issue is due to a call to the function `networkx.to_numpy_matrix` within the function `polytope.plot.plot_partition`. So avoiding this warning will be possible after the next release of the package `polytope`. (Note that inserting an `assert False` in a suitable line within the function `transition_directions_test` is not an alternative to passing the argument `-Werror`, because the `assert False` will result in a traceback where the `assert` statement appears, instead of a traceback that shows the call stack at the point where the warning was issued.) ## Speeding up debugging using `pytest` Also, since I had to be running `pytest` on the Python file `abstract_test.py`, `pytest` would collect all test functions, and run them. The file `abstract_test.py` happens to contain several slow test functions, so running them all just to observe the results for the one function of interest is not time-efficient. What I did to speed up runs was to rename all `test_*` functions contained in `abstract_test.py`, except for the one function of interest (namely `transition_directions_test`), to identifiers outside the patterns collected by `pytest`. A simpler alternative, for use with larger test files, is to do the opposite: rename only the function of interest to a different pattern, and then change the line `python_functions = ` in the configuration file `pytest.ini`. ## References - numpy/numpy#10142 (DEP: Pending deprecation warning for matrix) - numpy/numpy#10973 (DOC: advise against use of matrix) - scipy/scipy#8887 (MAINT: filter out np.matrix PendingDeprecationWarning's in numpy >=1.15) - scipy/scipy#9734 (PendingDeprecationWarning for np.matrix with pytest) - scikit-learn/scikit-learn#12327 (PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices) - scikit-learn/scikit-learn#13076 ([MRG] Ignore PendingDepWarnings of matrix subclass with pytest) - cvxpy/cvxpy#567 (NumPy matrix class is pending deprecation and issuing warnings) - cvxpy/cvxpy#637 (RF: Use a 2D np array instead of matrix to represent scalars) - cvxpy/cvxpy#638 (RF: Change np.matrix to np.array in several places) - cvxpy/cvxpy#644 (PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra) - https://docs.pytest.org/en/6.2.x/warnings.html#deprecationwarning-and-pendingdeprecationwarning
johnyf
added a commit
to tulip-control/tulip-control
that referenced
this pull request
Aug 20, 2021
because matrices have been deprecated in `numpy`. Usage of `scipy.sparse` is causing this issue when conversions to matrices are performed. I changed the: - calls to the method `scipy.sparse.lil.lil_matrix.todense` - to calls to the method `scipy.sparse.lil.lil_matrix.toarray`, to avoid the conversions that raise `PendingDeprecationWarning`s. The warnings are raised by the call to the method `lil_matrix.todense` because this call involves the instantiation of the class `numpy.matrix`, which is deprecated. In contrast, the method `lil_matrix.toarray` creates an instance of `numpy.ndarray`. (In fact, in the module `tulip.abstract.discretization`, wherever the method `lil_matrix.todense` was called, the value that it returned was immediately converted to a `numpy.ndarray`. So calling the method `lil_matrix.toarray` is actually more efficient.) The class `numpy.matrix` is deprecated and will probably be removed in the future. This will happen after arrangements have been made for `scipy.space`. (For these points and more information, read the references listed at the end.) Still, I do not think that continuing to use `scipy.sparse.lil_matrix` until when `numpy` removes matrices is a safe approach. Instead, using `numpy.ndarray` would be safer. ## Diagnosis I describe below the approach I (eventually) followed to debug this warning, because finding the cause was difficult. The issue is a `PendingDeprecationWarning` issued from `numpy`. This warning is visible in `pytest` runs, but *not* when running the Python test file directly. Moreover, `pytest` reports the warning, and from which test function the warning originates. The warning itself reads (I have wrapped the lines here): ``` ===================================== warnings summary ====================================== abstract_test.py::transition_directions_test /.../.virtualenvs/.../lib/python3.9/site-packages/numpy/matrixlib/defmatrix.py:69: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. return matrix(data, dtype=dtype, copy=False) ``` So the line in `tulip` that causes the warning cannot be found from the information contained in the warning. The above `PendingDeprecationWarning` was introduced in `numpy` in commit: numpy/numpy@11e9d2a This warning was then ignored in the module `scipy.sparse.__init__`, in `scipy` commit: scipy/scipy@a874bd5 It appears that this configuration of warnings by `scipy` interacts with `pytest` complexly: - when running with `pytest abstract_test.py`, the `PendingDeprecationWarning` is visible, but - when running with `python -X dev -- abstract_test.py`, the `PendingDeprecationWarning` is invisible. This behavior is due to the call: ```python warnings.filterwarnings( 'ignore', message='the matrix subclass is not the recommended way') ``` within `scipy.sparse.__init__.py` (introduced in the `scipy` commit that was mentioned above). Read also: https://docs.python.org/3/library/exceptions.html#PendingDeprecationWarning https://www.python.org/dev/peps/pep-0565/ As a result, it is difficult to find the cause within `tulip` of this `PendingDeprecationWarning`. ## Getting a traceback The test function that triggered the `PendingDeprecationWarning` from `numpy` was not failing, so there was no traceback that would indicate which line in `tulip` caused the warning. In addition, there was an earlier warning issued by `matplotlib`. So turning warnings to errors with the argument `-Werror` would cause `pytest` to turn the `matplotlib` warning into an error, and stop before the warning of interest: ```shell pytest -Werror abstract_test.py ``` So first I removed the `matplotlib` warnings (temporarily), by commenting the line `matplotlib.use('Agg')` in the file `abstract_test.py`. This made the warning of interest to become the first warning. I then passed `-Werror` to `pytest`, and this turned the `numpy` warning into an error, which produced the traceback shown below: (The cause of these `matplotlib` warnings (there are two) is in the package `polytope`, and has been addressed there, in commit: tulip-control/polytope@c464818 These changes will become available to `tulip` with the next `polytope` release. Until then, the CI tests of `tulip` will raise these `matplotlib` warnings. These warnings could be explicitly ignored by using `with pytest.warns`.) (The paths to `tulip` in the traceback below lead to the repository's `tulip`, instead of a directory under Python's `site-packages`, because during this phase of debugging I installed `tulip` with `pip install -e .`, to iterate faster while debugging.) ``` ../tulip/abstract/discretization.py:1666: in discretize_switched plot_mode_partitions(merged_abstr, show_ts, only_adjacent) ../tulip/abstract/discretization.py:1673: in plot_mode_partitions axs = swab.plot(show_ts, only_adjacent) ../tulip/abstract/discretization.py:187: in plot ax = ab.plot(show_ts, only_adjacent, color_seed) ../tulip/abstract/discretization.py:403: in plot ax = _plot_abstraction(self, show_ts, only_adjacent, ../tulip/abstract/discretization.py:446: in _plot_abstraction ax = ab.ppp.plot( ../tulip/abstract/prop2partition.py:600: in plot return plot_partition( .../.virtualenvs/.../lib/python3.9/site-packages/polytope/plot.py:90: in plot_partition trans = nx.to_numpy_matrix(trans, nodelist=ppp2trans) .../.virtualenvs/.../lib/python3.9/site-packages/networkx/convert_matrix.py:553: in to_numpy_matrix M = np.asmatrix(A, dtype=dtype) .../.virtualenvs/.../lib/python3.9/site-packages/numpy/matrixlib/defmatrix.py:69: in asmatrix return matrix(data, dtype=dtype, copy=False) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ subtype = <class 'numpy.matrix'> data = array([[1., 0., 0., 0., 0., 0.], [0., 1., 1., 0., 1., 1.], [1., 0., 1., 1., 0., 1.], [1., 0., 0., 1., 0., 0.], [0., 0., 0., 0., 1., 1.], [1., 0., 0., 0., 0., 1.]]) dtype = None, copy = False def __new__(subtype, data, dtype=None, copy=True): > warnings.warn('the matrix subclass is not the recommended way to ' 'represent matrices or deal with linear algebra (see ' 'https://docs.scipy.org/doc/numpy/user/' 'numpy-for-matlab-users.html). ' 'Please adjust your code to use regular ndarray.', PendingDeprecationWarning, stacklevel=2) E PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. .../.virtualenvs/.../lib/python3.9/site-packages/numpy/matrixlib/defmatrix.py:116: PendingDeprecationWarning ``` As the traceback shows, the issue is due to a call to the function `networkx.to_numpy_matrix` within the function `polytope.plot.plot_partition`. So avoiding this warning will be possible after the next release of the package `polytope`. (Note that inserting an `assert False` in a suitable line within the function `transition_directions_test` is not an alternative to passing the argument `-Werror`, because the `assert False` will result in a traceback where the `assert` statement appears, instead of a traceback that shows the call stack at the point where the warning was issued.) ## Speeding up debugging using `pytest` Also, since I had to be running `pytest` on the Python file `abstract_test.py`, `pytest` would collect all test functions, and run them. The file `abstract_test.py` happens to contain several slow test functions, so running them all just to observe the results for the one function of interest is not time-efficient. Running a single test function using `pytest` is possible by writing: ```shell pytest abstract_test.py::name_of_function ``` ## References - numpy/numpy#10142 (DEP: Pending deprecation warning for matrix) - numpy/numpy#10973 (DOC: advise against use of matrix) - scipy/scipy#8887 (MAINT: filter out np.matrix PendingDeprecationWarning's in numpy >=1.15) - scipy/scipy#9734 (PendingDeprecationWarning for np.matrix with pytest) - scikit-learn/scikit-learn#12327 (PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices) - scikit-learn/scikit-learn#13076 ([MRG] Ignore PendingDepWarnings of matrix subclass with pytest) - cvxpy/cvxpy#567 (NumPy matrix class is pending deprecation and issuing warnings) - cvxpy/cvxpy#637 (RF: Use a 2D np array instead of matrix to represent scalars) - cvxpy/cvxpy#638 (RF: Change np.matrix to np.array in several places) - cvxpy/cvxpy#644 (PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra) - https://docs.pytest.org/en/6.2.x/warnings.html#deprecationwarning-and-pendingdeprecationwarning
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the first commit of #10142, which just adds advice against using
matrix. I thought it was relatively uncontroversial, and should not cause any test failures, so probably good to treat separately.