DEPR: Deprecate NDFrame.as_matrix #18458

topper-123 · 2017-11-23T22:21:30Z

[x ] xref DEPR: let's deprecate #18262
[x ] passes git diff upstream/master -u -- "*.py" | flake8 --diff
[ x] whatsnew entry

Deprecating NDFrame.as_matrix as per discussion in #18262.

codecov · 2017-11-24T04:23:36Z

Codecov Report

Merging #18458 into master will increase coverage by 0.02%.
The diff coverage is 85.71%.

@@            Coverage Diff             @@
##           master   #18458      +/-   ##
==========================================
+ Coverage    91.3%   91.32%   +0.02%     
==========================================
  Files         163      163              
  Lines       49781    49783       +2     
==========================================
+ Hits        45451    45464      +13     
+ Misses       4330     4319      -11

Flag	Coverage Δ
#multiple	`89.12% <85.71%> (+0.02%)`	⬆️
#single	`40.72% <64.28%> (-0.01%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/panel.py	`96.85% <0%> (-0.29%)`	⬇️
pandas/core/internals.py	`94.47% <100%> (ø)`	⬆️
pandas/core/generic.py	`95.78% <83.33%> (+0.05%)`	⬆️
pandas/plotting/_converter.py	`65.25% <0%> (+1.81%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 38f41e6...48e1fc8. Read the comment docs.

jorisvandenbossche

Thanks for the PR! Left some comments

jorisvandenbossche · 2017-11-24T08:55:56Z

doc/source/whatsnew/v0.22.0.txt

@@ -82,7 +82,7 @@ Deprecations
 ~~~~~~~~~~~~

 - ``Series.from_array`` and ``SparseSeries.from_array`` are deprecated. Use the normal constructor ``Series(..)`` and ``SparseSeries(..)`` instead (:issue:`18213`).
-
+- ``NDFrame.as_matrix`` is deprecated. Use ``NDFrame.values`` instead (:issue:`18458`).


Can you use DataFrame instead of NDFrame (basically NDFrame should never show up in the docs, it is an internal implementation detail, although there are some places where it leaks through ..)

jorisvandenbossche · 2017-11-24T08:58:08Z

pandas/core/generic.py

@@ -3735,6 +3735,9 @@ def _get_bool_data(self):

    def as_matrix(self, columns=None):
        """
+        DEPRECATED: This method will be removed in a future version.
+        Use :meth:`NDFrame.values` instead.


same comment here

jorisvandenbossche · 2017-11-24T09:00:13Z

pandas/core/generic.py

@@ -3770,6 +3773,8 @@ def as_matrix(self, columns=None):
        --------
        pandas.DataFrame.values
        """
+        warnings.warn("This method will be removed in a future version. "
+                      "Use ``.values`` instead.")


This needs to be a FutureWarning, and you also need to set a stacklevel (this is a top-level method, stacklevel=2 should be the correct one)

Can you also mention as_matrix explicitly in the message (that is clearer as you don't always directly see what line or what method on a certain line is causing the warning).
And also follow a bit the typical message like "'as_matrix' is deprecated and will be removed in a future version. Use .."

jorisvandenbossche · 2017-11-24T09:07:11Z

pandas/tests/frame/test_api.py

@@ -243,9 +243,9 @@ def test_itertuples(self):
    def test_len(self):
        assert len(self.frame) == len(self.frame.index)

-    def test_as_matrix(self):
+    def test_values(self):


Can you leave one test (or now add a test_as_matrix_deprecated that copies eg the first case of this test) that uses as_matrix and that asserts it raises a warning and is the same as .values

Ok. I've added in below test_repr_with_mi_nat

jorisvandenbossche · 2017-11-24T12:37:28Z

General question: I am wondering if we shouldn't rather recommend to do np.asarray(df) instead of df.values ?

topper-123 · 2017-11-24T13:05:17Z

Interesting, does it conform to .values in all cases?.

Anyway, I suggest that that will be a separate issue from this one.

topper-123 · 2017-11-24T13:52:30Z

I've changed the PR according to the comments. Is this ok?

jreback · 2017-11-24T19:52:30Z

pandas/core/generic.py

@@ -3791,7 +3796,10 @@ def values(self):
        int32. By numpy.find_common_type convention, mixing int64 and uint64
        will result in a flot64 dtype.
        """
-        return self.as_matrix()


just return .values

This is the def of .values

oh, then we should push this entirely down to the BlockManager, so pass the axis as well. The BM can then do the transpose, we don't like to do things in user code like this.

We should actually deprecate .values as well, in favor of .to_numpy() which is the future spelling anyhow.

This is already almost entirely pushed down, since it is calling directly BlockManager.as_matrix (but OK, the transpose could be done inside BlockManager.as_matrix). The consolidate_inplace needs to happens in NDFrame I think.

Deprecating .values is a no-go IMO, but let's deprecate that in another issue if you want

oh, then we should push this entirely down to the BlockManager, so pass the axis as well. The BM can then do the transpose, we don't like to do things in user code like this.

I'm not too familiar with the BlockManager, except accessing it with ._data. How do I pass the axis to it, and how do I transpose in a BlockManager? E.g. it doesn't have a .transpose method.

look at _take in pandas/core/generic.py, you just pass the get_block_manager_axis to it.

then add axis= to def as_matrix in pandas/core/internals and do the transpose there.

The reason for pushing this to internals is to avoid all of the wrapping layers (e.g. frame/series, etc.) to know about internal details like this.

as_matrix doesn't have a axis parameter. Do you mean adding a new axis parameter there and transposing inside, if axis=0 (where I assume from my limited knowledge that BlockManager.axes[0] is always the same as dataFrame.axes[1], correct?)

Hi @jreback, I've tried quite a bit today and I can't push this down while getting the tests to pass.

Could you look at my latest commit (not passing ATM) and give some advice?

jreback · 2017-11-24T19:52:48Z

pandas/tests/frame/test_api.py

@@ -369,6 +369,12 @@ def test_values(self):
        self.frame.values[:, 0] = 5.
        assert (self.frame.values[:, 0] == 5).all()

+    def test_as_matrix_deprecated(self):
+        with tm.assert_produces_warning(FutureWarning):


add the issue number

jreback · 2017-11-25T23:06:32Z

pandas/core/internals.py

        if len(self.blocks) == 0:
            return np.empty(self.shape, dtype=float)

+        other_axis = abs(axis-1)


don't do this

jreback · 2017-11-25T23:06:38Z

pandas/core/internals.py

        if items is not None:
-            mgr = self.reindex_axis(items, axis=0)
+            mgr = self.reindex_axis(items, axis=other_axis)


leave this alone

jreback · 2017-11-25T23:09:09Z

pandas/core/generic.py

-        return self.as_matrix()
+        self._consolidate_inplace()
+        bm_axis = self._get_block_manager_axis(axis=1)
+        return self._data.as_matrix(axis=bm_axis)


instead of this, just pass transpose = self._AXIS_REVERSED in.

But _data.as_matrix has no transpose parameter, do you mean axis=self._AXIS_REVERSED?

no I mean add the argument transpose=False rather than axis prob simpler for now.

jreback · 2017-11-25T23:09:28Z

pandas/core/internals.py

        else:
            mgr = self

        if self._is_single_block or not self.is_mixed_type:
-            return mgr.blocks[0].get_values()
+            arr = mgr.blocks[0].get_values()
        else:


change this to transpose instead. should be straightforward from here.

Thanks a lot, transpose is of course much better than axis.

The issue was actually in the if len(self.blocks) == 0: block, as the empty array also must be transposed.

Everything is green now locally and I've pushed that upstream.

topper-123 · 2017-11-26T00:11:25Z

A thought: BlockManager.as_matrix is a strange name as it implies a np.matrix is returned.

Should I not just change it to BlockManager.as_array? Or, do any external libraries rely on calling BlockManager directly, e.g. statsmodel?

jreback · 2017-11-26T00:12:24Z

@topper-123 ok this lgtm. ping on green.

jreback · 2017-11-26T00:13:20Z

Should I not just change it to BlockManager.as_array? Or, do any external libraries rely on calling BlockManager directly, e.g. statsmodel?

this is completely private and internal. as_array would be fine. while you are at it, can you add a doc-string.

ping when ready.

jreback · 2017-11-26T00:28:17Z

pandas/core/generic.py

        self._consolidate_inplace()
        if self._AXIS_REVERSED:
-            return self._data.as_matrix(columns).T


this should just return self.values

Returning self.values implies that columns=None which isn't necessary true for user code.

jreback · 2017-11-26T00:28:42Z

pandas/core/generic.py

@@ -3842,7 +3848,7 @@ def as_blocks(self, copy=True):
        .. deprecated:: 0.21.0

        NOTE: the dtypes of the blocks WILL BE PRESERVED HERE (unlike in
-              as_matrix)
+              as_array)


leave this spelling, this is a top-level method name

jreback · 2017-11-26T00:28:56Z

pandas/core/internals.py

@@ -3670,19 +3670,22 @@ def copy(self, deep=True, mgr=None):
        return self.apply('copy', axes=new_axes, deep=deep,
                          do_integrity_check=False)

-    def as_matrix(self, items=None):
+    def as_array(self, transpose=False, items=None):
        if len(self.blocks) == 0:


can you add a doc-string

jreback · 2017-11-26T00:41:46Z

pandas/core/generic.py

@@ -3770,10 +3773,12 @@ def as_matrix(self, columns=None):
        --------
        pandas.DataFrame.values
        """
+        warnings.warn("method ``as_matrix`` will be removed in a future version. "
+                      "Use ``values`` instead.", FutureWarning, stacklevel=2)
        self._consolidate_inplace()
        if self._AXIS_REVERSED:


ok so this should be the same as below (e.g. passing transpose=). make sure we still have a test on this (e.g. that passes columns).

Ok, and test_as_matrix_deprecated has been modified to a take columns param.

jreback · 2017-11-26T01:02:53Z

pandas/core/generic.py

@@ -3770,10 +3773,11 @@ def as_matrix(self, columns=None):
        --------
        pandas.DataFrame.values
        """
+        warnings.warn("method ``as_matrix`` will be removed in a future version. "


"method method .as_matrix()...."
"Use .values instead"

I not sure I understand, assume you want the backticks gone. Change uploaded.

topper-123 · 2017-11-26T14:39:47Z

All green, @jreback

jreback · 2017-11-26T15:04:08Z

thanks @topper-123

as_matrix() got removed with Pandas 1.0.0 (pandas-dev/pandas#18458), replaced with values

See GH18458（pandas-dev/pandas#18458). Since as_matrix is deprecated for pd.DataFrame and pd.Series, use data.values instead.

Pandas.DataFrame.as_matrix is deprecated since version 0.23.0 (see docs and PR 18458). According to the documentation for Pandas 0.25.1, the recommended function is DataFrame.to_numpy() in place of DataFrame.values or DataFrame.as_matrix(). pandas-dev/pandas#18458 https://pandas.pydata.org/pandas-docs/version/0.23/generated/pandas.DataFrame.as_matrix.html https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.values.html

topper-123 force-pushed the deprecate_as_matrix branch 2 times, most recently from d981cb8 to 7fd4b71 Compare November 24, 2017 00:34

jorisvandenbossche added the Deprecate Functionality to remove in pandas label Nov 24, 2017

jorisvandenbossche added this to the 0.22.0 milestone Nov 24, 2017

jsexauer mentioned this pull request Nov 24, 2017

DEPR: Clean up list of deprecations from prior versions #6581

Closed

1 task

jorisvandenbossche reviewed Nov 24, 2017

View reviewed changes

topper-123 force-pushed the deprecate_as_matrix branch from c5ef79b to ae8dc5a Compare November 24, 2017 12:52

jreback mentioned this pull request Nov 24, 2017

DEPR: let's deprecate #18262

Closed

34 tasks

jreback requested changes Nov 24, 2017

View reviewed changes

topper-123 force-pushed the deprecate_as_matrix branch from ae8dc5a to 9255589 Compare November 24, 2017 21:26

Deprecate NDFrame.as_matrix

6b71d4a

topper-123 force-pushed the deprecate_as_matrix branch 5 times, most recently from 61c5a2d to 579d128 Compare November 25, 2017 22:46

jreback requested changes Nov 25, 2017

View reviewed changes

topper-123 force-pushed the deprecate_as_matrix branch from 579d128 to eed2fb6 Compare November 26, 2017 00:00

updating according to comments

0d47e46

topper-123 force-pushed the deprecate_as_matrix branch from eed2fb6 to 0d47e46 Compare November 26, 2017 00:14

jreback requested changes Nov 26, 2017

View reviewed changes

topper-123 force-pushed the deprecate_as_matrix branch 2 times, most recently from 0816d86 to 6ad82de Compare November 26, 2017 00:33

topper-123 force-pushed the deprecate_as_matrix branch from 6ad82de to e48ea09 Compare November 26, 2017 00:36

jreback reviewed Nov 26, 2017

View reviewed changes

topper-123 force-pushed the deprecate_as_matrix branch 2 times, most recently from eb0b82d to e1066d2 Compare November 26, 2017 01:00

jreback reviewed Nov 26, 2017

View reviewed changes

topper-123 force-pushed the deprecate_as_matrix branch 5 times, most recently from d8b4d0b to a343fe3 Compare November 26, 2017 11:25

rename BlockManager.as_matrix to .as_array

48e1fc8

topper-123 force-pushed the deprecate_as_matrix branch from a343fe3 to 48e1fc8 Compare November 26, 2017 12:02

jreback approved these changes Nov 26, 2017

View reviewed changes

jreback merged commit f26bed6 into pandas-dev:master Nov 26, 2017

topper-123 deleted the deprecate_as_matrix branch November 26, 2017 15:07

jeremiahpslewis mentioned this pull request May 16, 2018

Fix pandas 0.23.0 deprecation of as_matrix shap/shap#87

Merged

jreback mentioned this pull request Nov 20, 2019

DEPR: deprecations log for removed issues #13777

Closed

giantZorg added a commit to giantZorg/multivariate_time_series_forecasting that referenced this pull request Feb 4, 2020

as_matrix replaced

8e8bdd9

as_matrix() got removed with Pandas 1.0.0 (pandas-dev/pandas#18458), replaced with values

giantZorg mentioned this pull request Feb 4, 2020

as_matrix replaced opringle/multivariate_time_series_forecasting#2

Open

atangfan added a commit to atangfan/Deep-Portfolio-Management that referenced this pull request May 11, 2020

Replace pd.as_matrix by pd.values

36ac2c0

See GH18458（pandas-dev/pandas#18458). Since as_matrix is deprecated for pd.DataFrame and pd.Series, use data.values instead.

mcrowson mentioned this pull request Jul 1, 2020

pandas deprecated to_matrix Kismuz/btgym#139

Merged

jnirschl mentioned this pull request Aug 10, 2020

Update Deprecated Pandas Function Call ustunb/risk-slim#9

Merged

xaf-cv mentioned this pull request Jan 26, 2022

.as_matrix() -> .to_numpy() hpatches/hpatches-benchmark#44

Open

DEPR: Deprecate NDFrame.as_matrix #18458

DEPR: Deprecate NDFrame.as_matrix #18458

Conversation

topper-123 commented Nov 23, 2017

codecov bot commented Nov 24, 2017 • edited

Codecov Report

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorisvandenbossche commented Nov 24, 2017

topper-123 commented Nov 24, 2017

topper-123 commented Nov 24, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

topper-123 Nov 24, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

topper-123 Nov 25, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

topper-123 Nov 26, 2017 • edited

Choose a reason for hiding this comment

topper-123 commented Nov 26, 2017 • edited

jreback commented Nov 26, 2017

jreback commented Nov 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

topper-123 commented Nov 26, 2017

jreback commented Nov 26, 2017

codecov bot commented Nov 24, 2017 •

edited

topper-123 Nov 24, 2017 •

edited

topper-123 Nov 25, 2017 •

edited

topper-123 Nov 26, 2017 •

edited

topper-123 commented Nov 26, 2017 •

edited