DEPR: Series ndarray properties (strides, data, base, itemsize, flags) #20721

jorisvandenbossche · 2018-04-17T14:50:25Z

closes Deprecate Series.strides, .base, .data, .itemsize, .flags (numpy) attributes ? #20419
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

jschendel · 2018-04-17T23:31:46Z

pandas/core/base.py

@@ -737,11 +737,17 @@ def item(self):
    @property
    def data(self):
        """ return the data pointer of the underlying data """
+        warnings.warn("Series/Index.data is deprecated and will be "


Might be nice to make the warning messages a bit more dynamic with something like:

"{obj}.data is deprecated...".format(obj=type(self).__name__)

Yes, thank you (somehow I was the docstring mindset thinking we needed to adapt this upon class definition, but of course this can be done dynamically on run time :-))

pep8speaks · 2018-04-18T08:43:41Z

Hello @jorisvandenbossche! Thanks for updating the PR.

Cheers ! There are no PEP8 issues in this Pull Request. 🍻

Comment last updated on April 24, 2018 at 12:42 Hours UTC

codecov · 2018-04-18T08:43:57Z

Codecov Report

Merging #20721 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #20721      +/-   ##
==========================================
- Coverage   91.85%   91.84%   -0.01%     
==========================================
  Files         153      153              
  Lines       49308    49316       +8     
==========================================
+ Hits        45290    45296       +6     
- Misses       4018     4020       +2

Flag	Coverage Δ
#multiple	`90.24% <100%> (-0.01%)`	⬇️
#single	`41.89% <25%> (-0.01%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/algorithms.py	`94.37% <100%> (ø)`	⬆️
pandas/core/internals.py	`95.58% <100%> (ø)`	⬆️
pandas/core/base.py	`96.83% <100%> (+0.03%)`	⬆️
pandas/core/indexes/category.py	`97.03% <0%> (-0.27%)`	⬇️
pandas/core/arrays/categorical.py	`95.64% <0%> (-0.14%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0ae7e90...feb7caf. Read the comment docs.

jorisvandenbossche · 2018-04-18T11:25:53Z

pandas/tests/groupby/aggregate/test_other.py

@@ -328,7 +328,7 @@ def test_series_agg_multi_pure_python():
         'F': np.random.randn(11)})

    def bad(x):
-        assert (len(x.base) > 0)
+        assert (len(x.values.base) > 0)


Not really sure what the purpose of this assert actually is (was introduced in 71e9046)

jreback

pls run all tests and see if any warnings comes up

jreback · 2018-04-19T10:38:03Z

pandas/core/indexes/datetimelike.py

+        """ return the base object if the memory of the underlying data is
+        shared
+        """
+        # override deprecated property in IndexOpsMixin, as we still need


your explanation doesn't make sense, we are deprecating these no? you need to change the way its accessed in the code to remove the warnings.

The explanation perfectly makes sense, but you don't like the way I solved it I suppose (I agree it is not the nicest, but I was thinking that this would only be temporarily until DatetimeTZBlock is an ExtensionBlock).

To fix the usage itself, .base is used in two placed:

Block.is_view: this I can override in DatetimeTzBlock to check self.values.values.base

concatenate_join_units:

pandas/pandas/core/internals.py

Line 5544 in ede11af

if copy and concat_values.base is not None:

Not fully sure about this one what to do here.

my point is this method should not exist (as you are ddeprecateding), and would rather have you fix the usage

As I said, I agree with that. Do you have a suggestion for the second case? (inside concatenate_join_units) In #20745 I need to touch the same code, and checked if the values have a base attribute, but that also feels hacky

yes, see my comment there, only check base if its an ndarray

jreback · 2018-04-19T15:46:13Z

pandas/core/algorithms.py

@@ -1491,6 +1491,8 @@ def take_nd(arr, indexer, axis=0, out=None, fill_value=np.nan, mask_info=None,
    if is_sparse(arr):
        arr = arr.get_values()

+    arr = np.array(arr, copy=False)


np.asarray slightly more idiomatic

Ah, yes (for some reason I wanted the copy=False keyword, but for asarray that is of course just the default behaviour)

jreback · 2018-04-19T15:46:37Z

pandas/core/indexes/datetimelike.py

+        """ return the base object if the memory of the underlying data is
+        shared
+        """
+        # override deprecated property in IndexOpsMixin, as we still need


my point is this method should not exist (as you are ddeprecateding), and would rather have you fix the usage

jreback · 2018-04-21T16:45:25Z

can you update

…ay-attrs

jorisvandenbossche · 2018-04-23T07:46:30Z

Now that #20745 is merged, updated this.

TomAugspurger · 2018-04-23T19:02:12Z

@jorisvandenbossche a couple new warnings I think: https://travis-ci.org/pandas-dev/pandas/jobs/369977719#L2762

pandas/tests/indexes/datetimes/test_datetimelike.py::TestDatetimeIndex::()::test_copy_name
  /Users/travis/build/pandas-dev/pandas/pandas/core/indexes/base.py:551: FutureWarning: DatetimeIndex.base is deprecated and will be removed in a future version
    orig = orig if orig.base is None else orig.base
pandas/tests/internals/test_internals.py::TestBlockManager::()::test_copy
  /Users/travis/build/pandas-dev/pandas/pandas/tests/internals/test_internals.py:469: FutureWarning: DatetimeIndex.base is deprecated and will be removed in a future version
    assert cp_blk.values.base is blk.values.base
  /Users/travis/build/pandas-dev/pandas/pandas/tests/internals/test_internals.py:477: FutureWarning: DatetimeIndex.base is deprecated and will be removed in a future version
    if cp_blk.values.base is not None and blk.values.base is not None:
  /Users/travis/build/pandas-dev/pandas/pandas/tests/internals/test_internals.py:478: FutureWarning: DatetimeIndex.base is deprecated and will be removed in a future version
    assert cp_blk.values.base is not blk.values.base

Can you see if they're easy to fix?

jreback · 2018-04-24T10:03:20Z

lgtm, other than @TomAugspurger comments.

jreback · 2018-04-24T12:46:12Z

lgtm. modulo any warnings fixes. note that I just pushed a CI fix for lint (on other PR)

jorisvandenbossche · 2018-04-24T13:13:59Z

pandas/core/indexes/base.py

@@ -548,6 +548,9 @@ def _deepcopy_if_needed(self, orig, copy=False):
        """
        if copy:
            # Retrieve the "base objects", i.e. the original memory allocations
+            if not isinstance(orig, np.ndarray):
+                # orig is a DatetimeIndex
+                orig = orig.values


@jreback This special case is because we sometimes construct DatetimeIndex objects where the _data attribute still is a DatetimeIndex in itself.
To me this seems like something we should not do? (it doesn't hurt, but there is also no reason to have it I think, so better always store the ndarray to be consistent?)

we should never do that
how did this come up?

Just opened an issue about it: #20810
It came up from fixing the warnings in the tests. The above case, where orig is not always a ndarray (hence my special case if statement), triggered it, because orig always comes from a _data attribute, and so I expected it to always be a ndarray.

TomAugspurger · 2018-04-25T21:07:26Z

Thanks @jorisvandenbossche

naazneen · 2020-05-23T14:54:48Z

I was doing scipy.fftpack.fft over a series and it showed error as
Attriute error: Series object has no attribute flags.

What should I use instead? And how to overcome it?

DEPR: Series ndarray properties (strides, data, base, itemsize, flags)

c3ac291

jorisvandenbossche added the Deprecate Functionality to remove in pandas label Apr 17, 2018

jorisvandenbossche added this to the 0.23.0 milestone Apr 17, 2018

jschendel reviewed Apr 17, 2018

View reviewed changes

better warning message

11182b1

jorisvandenbossche force-pushed the depr-series-ndarray-attrs branch from f10dd3f to 11182b1 Compare April 18, 2018 08:44

jorisvandenbossche added 2 commits April 18, 2018 10:47

add whatsnew

1aeaf2e

fix usage in tests

5339975

jorisvandenbossche commented Apr 18, 2018

View reviewed changes

jreback requested changes Apr 19, 2018

View reviewed changes

jsexauer mentioned this pull request Apr 19, 2018

DEPR: Clean up list of deprecations from prior versions #6581

Closed

1 task

fix remaining tests

c78d5d6

jorisvandenbossche mentioned this pull request Apr 19, 2018

Fix pd.merge to preserve ExtensionArrays dtypes #20745

Merged

jreback requested changes Apr 19, 2018

View reviewed changes

jorisvandenbossche added 3 commits April 23, 2018 09:38

Merge remote-tracking branch 'upstream/master' into depr-series-ndarr…

5bd2940

…ay-attrs

also deprecate DatetimeIndex.base

160ace0

array -> asarray

d2088d2

jreback mentioned this pull request Apr 23, 2018

pandas.DatetimeIndex.data and pandas.TimedeltaIndex.data fail with "ValueError: cannot include dtype 'M' in a buffer" #20788

Closed

jorisvandenbossche added 2 commits April 24, 2018 14:33

fix usage of base in DatetimeIndexConstructor (_deepcopy_if_needed)

7687330

fix usage of .base in BlockManager test_copy

feb7caf

jreback approved these changes Apr 24, 2018

View reviewed changes

jorisvandenbossche commented Apr 24, 2018

View reviewed changes

jorisvandenbossche mentioned this pull request Apr 24, 2018

BUG: inconsistent state of DatetimeIndex._data #20810

Closed

TomAugspurger merged commit 21e884f into pandas-dev:master Apr 25, 2018

jschendel mentioned this pull request May 2, 2018

CLN: Fix typo in deprecation warning - "strudes" --> "strides" #20929

Merged

jorisvandenbossche deleted the depr-series-ndarray-attrs branch July 24, 2018 08:53

jorisvandenbossche mentioned this pull request Jul 24, 2018

REF: IntervalIndex[IntervalArray] #20611

Merged

jreback mentioned this pull request Nov 30, 2019

DEPR: deprecations log for removed issues #13777

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DEPR: Series ndarray properties (strides, data, base, itemsize, flags) #20721

DEPR: Series ndarray properties (strides, data, base, itemsize, flags) #20721

jorisvandenbossche commented Apr 17, 2018

jschendel Apr 17, 2018

jorisvandenbossche Apr 18, 2018

pep8speaks commented Apr 18, 2018 •

edited

Loading

codecov bot commented Apr 18, 2018 •

edited

Loading

jorisvandenbossche Apr 18, 2018

jreback left a comment

jreback Apr 19, 2018

jorisvandenbossche Apr 19, 2018 •

edited

Loading

jreback Apr 19, 2018

jorisvandenbossche Apr 19, 2018

jreback Apr 19, 2018

jreback Apr 19, 2018

jorisvandenbossche Apr 19, 2018

jreback Apr 19, 2018

jreback commented Apr 21, 2018

jorisvandenbossche commented Apr 23, 2018

TomAugspurger commented Apr 23, 2018

jreback commented Apr 24, 2018

jreback commented Apr 24, 2018

jorisvandenbossche Apr 24, 2018

jreback Apr 24, 2018

jorisvandenbossche Apr 24, 2018

TomAugspurger commented Apr 25, 2018

naazneen commented May 23, 2020

DEPR: Series ndarray properties (strides, data, base, itemsize, flags) #20721

DEPR: Series ndarray properties (strides, data, base, itemsize, flags) #20721

Conversation

jorisvandenbossche commented Apr 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pep8speaks commented Apr 18, 2018 • edited Loading

Comment last updated on April 24, 2018 at 12:42 Hours UTC

codecov bot commented Apr 18, 2018 • edited Loading

Codecov Report

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorisvandenbossche Apr 19, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Apr 21, 2018

jorisvandenbossche commented Apr 23, 2018

TomAugspurger commented Apr 23, 2018

jreback commented Apr 24, 2018

jreback commented Apr 24, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TomAugspurger commented Apr 25, 2018

naazneen commented May 23, 2020

pep8speaks commented Apr 18, 2018 •

edited

Loading

codecov bot commented Apr 18, 2018 •

edited

Loading

jorisvandenbossche Apr 19, 2018 •

edited

Loading