ENH: Add isocalendar accessor to DatetimeIndex and Series.dt #33220

mgmarino · 2020-04-01T20:35:15Z

This PR adds the the isocalendar property to DatetimeIndex and the corresponding Series.dt accessor. The property returns a DataFrame, e.g.:

>>> idx = pd.date_range(start='2019-12-29', freq='D', periods=4)
>>> idx.isocalendar
   year  week  day
0  2019    52    7
1  2020     1    1
2  2020     1    2
3  2020     1    3

and

>>> pandas.to_datetime(pandas.Series(["2020-01-01"])).dt.isocalendar
   year  week  day
0  2020     1    
>>> pandas.to_datetime(pandas.Series(["2019-12-31"])).dt.isocalendar.week
0    1
Name: week, dtype: int32

The behavior is consistent with Timestamp.isocalendar and datetime.date.isocalendar.

For more information about ISO 8601 calendar, see e.g. https://en.wikipedia.org/wiki/ISO_week_date.

Address GH33206

closes ENH: Add accessor for isocalendar returning iso year, week, and day #33206
tests added / passed
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

Note that I am very happy to rename the field, but I wanted to go ahead and open up the PR.

WillAyd · 2020-04-01T22:21:27Z

Thanks for the PR. I agree the naming is a little weird, though do we need a dedicated property for this? Might be more natural to just convert to the appropriate period and still use the .year attribute therefrom

@jbrockmendel @mroeschke

mroeschke · 2020-04-01T22:44:48Z

Agreed with @WillAyd.

Since Timestamp has an isocalendar method from the standard library, I wouldn't be opposed to a DatetimeIndex.isocalendar or Series.dt.isocalendar method that converts the dates to returns a 3-tuple, (ISO year, ISO week number, ISO weekday).

mgmarino · 2020-04-02T06:58:59Z

@WillAyd @mroeschke Thanks for the comments!

I agree that the isocalendar method approach is cleaner and more symmetric with Timestamp, this was slightly easier for me to get my feet wet with the code. An additional advantage of this is that it makes it clearer that all values are calculated using ISO (8601).

Regarding whether this functionality is needed at all, I think if weekofyear is provided (which is currently the ISO week), then there absolutely should to be a way to get the year that corresponds to it.

mroeschke · 2020-04-02T16:34:09Z

I'd say if we include isocalendar (and think of a nice API to easily get the .year, .week, etc), we could deprecate weekofyear (which datetime.datetime doesn't have).

Generally, I believe we should align all .dt attributes/methods to what exists in datetime.datetime)

mgmarino · 2020-04-02T18:11:26Z

Yes, I think a series of tuples is not completely user friendly. What about like what's done in .dt.components here, i.e. it returns a frame?

isocalendar could be a property returning a frame (with three columns year, week, day), and then the respective series could simply be accessed like:

.dt.isocalendar.year
.dt.isocalendar.week
.dt.isocalendar.day

What do you think?

Re the deprecation of weekofyear, I think that would be a good idea. I get the impression that there's confusion around it by looking through old issues. By access week through isocalendar, it would I think reduce confusion.

mroeschke · 2020-04-02T18:32:25Z

+1 to that API @mgmarino. Interested in working on both the isocalendar addition and deprecation of weekofyear (and if there's additional attributes like this)?

mgmarino · 2020-04-02T18:38:19Z

Definitely! I'll need a bit of a hand with the Deprecation process, but I think I'm pretty clear how to implement isocalendar. The deprecation should probably be done in a separate PR anyways.

For the isocalendar changes, shall I just keep this PR (to have the thread), with an appropriate rename?

mroeschke · 2020-04-02T18:40:43Z

The deprecation should probably be done in a separate PR anyways.

Sure sounds good.

For the isocalendar changes, shall I just keep this PR (to have the thread), with an appropriate rename?

Definitely. Feel free to retool this PR (and original issue) about adding isocalendar

pandas/core/arrays/datetimes.py

mgmarino · 2020-04-04T14:06:57Z

FYI, the failing tests seem to be due to visual studio not liking the ctuple return. I’ll still need some time to investigate a fix and/or rewrite to avoid the ctuple.

mgmarino · 2020-04-04T19:36:38Z

@mroeschke

Ok, I'm pretty confident that the build will now succeed (it succeeded on my local Azure pipelines), but here's the news:

The error was due to a bug in cython fixed in 0.29.16 (See cimport does not work for functions returning tuples cython/cython#2745, cimported ctuples don’t compile their declaration cython/cython#1427)
Because of this, I had to bump the minimum version of cython to 0.29.16.

So, I would understand if this feature is not important enough to do this, so I'm happy to have your feedback, if I should:

leave as is,
bump the dependency version upgrade out to a separate PR.
rework the internal code to avoid using ctuple.
other?

Thanks for your help on this!

pandas/tests/series/test_datetime_values.py

mroeschke · 2020-04-06T00:30:41Z

I think it's okay to bump Cython since it's a build dependency cc @jbrockmendel.

mgmarino · 2020-04-06T13:34:37Z

Current comments addressed, failing doc build seems to be handled with #33309.

jbrockmendel · 2020-04-06T16:06:36Z

i think we're OK to bump cython to the newest bugfix release

mroeschke · 2020-04-06T16:36:30Z

pandas/core/indexes/accessors.py

+
+        Examples
+        --------
+        >>> pd.to_datetime(pd.Series(["2020-01-01"])).dt.isocalendar


I'd just create one datetime Series in the beginning and just reference that in the other examples:

>>> ser = pd.Series(pd.to_datetime(["2020-01-01", "NaT"])) >>> ser ... >>> ser.dt.isocalendar >>> ser.dt.isocalendar.week

mgmarino · 2020-04-08T06:57:07Z

Rebased, happy to hear any further comments!

jreback

some comments. can you add some prose in timeseries.rst about this (basically just showing it, add a versionadded 1.1 tag). ping on green.

jreback · 2020-04-08T15:28:13Z

pandas/core/arrays/datetimes.py

+
+        sarray = fields.build_isocalendar_sarray(self.asi8)
+        iso_calendar_df = DataFrame(
+            sarray, columns=["year", "week", "day"], dtype="Int64"


use UInt32 (as that's the default for all calendar component ops)

jreback · 2020-04-08T15:29:23Z

pandas/_libs/tslibs/fields.pyx

+    with nogil:
+        for i in range(count):
+            if dtindex[i] == NPY_NAT:
+                ret_val = -1, -1, -1


make these return 0

jreback · 2020-04-08T15:29:58Z

pandas/core/indexes/accessors.py

+        >>> ser.dt.isocalendar.week
+        0      53
+        1    <NA>
+        Name: week, dtype: Int64


you will need to update this as per the dtype change

mgmarino · 2020-04-08T19:33:21Z

@jreback Thanks for the comments, I finally understood it now.

Dtype -> UInt32, Doc added, Tests green.

jreback · 2020-04-08T19:40:24Z

lgtm. cc @TomAugspurger ok here?

jreback · 2020-04-10T21:04:50Z

this lgtm. let's merge on green.

mroeschke · 2020-04-12T21:34:58Z

Thanks @mgmarino! Very nice. Happy to have a follow up to deprecate weekofyear.

jorisvandenbossche · 2020-04-13T12:46:36Z

Since isocalendaris a method and not an attribute on Timestamp, should we make dt.isocalendar a method instead of a property as well for consistency?

mroeschke · 2020-04-13T16:13:50Z

Good point @jorisvandenbossche.

@mgmarino mind creating a follow up PR that makes isocalendar a method instead of an attribute?

For consistency with `Timestamp.isocalendar`, this should rather be a method. Followup of pandas-dev#33220, see the discussions following the merge

mgmarino · 2020-04-13T20:01:42Z

mind creating a follow up PR that makes isocalendar a method instead of an attribute?

@mroeschke Sure no problem, seems reasonable. The PR is open, I'll ping on green.

Happy to have a follow up to deprecate weekofyear.

Will do soon!

For consistency with `Timestamp.isocalendar`, this should rather be a method. Followup of pandas-dev#33220, see the discussions following the merge

For consistency with `Timestamp.isocalendar`, this should rather be a method. Followup of #33220, see the discussions following the merge

For consistency with `Timestamp.isocalendar`, this should rather be a method. Followup of pandas-dev#33220, see the discussions following the merge

mgmarino force-pushed the add-year-for-week-of-year branch 2 times, most recently from 6463c51 to 18b7fef Compare April 1, 2020 21:11

WillAyd added Needs Discussion Requires discussion from core team before further action Datetime Datetime data dtype labels Apr 1, 2020

mgmarino force-pushed the add-year-for-week-of-year branch from 18b7fef to dc02967 Compare April 3, 2020 20:36

mgmarino changed the title ~~ENH: Add year for week of year~~ ENH: Add isocalendar accessor to DatetimeIndex and Series.dt Apr 3, 2020

mgmarino mentioned this pull request Apr 3, 2020

ENH: Add accessor for isocalendar returning iso year, week, and day #33206

Closed

mgmarino force-pushed the add-year-for-week-of-year branch 2 times, most recently from bd86caf to 90cb3c1 Compare April 3, 2020 21:48

mroeschke reviewed Apr 3, 2020

View reviewed changes

pandas/core/arrays/datetimes.py Outdated Show resolved Hide resolved

mroeschke added Enhancement and removed Needs Discussion Requires discussion from core team before further action labels Apr 4, 2020

mroeschke reviewed Apr 6, 2020

View reviewed changes

pandas/tests/series/test_datetime_values.py Outdated Show resolved Hide resolved

mgmarino force-pushed the add-year-for-week-of-year branch from a0bff50 to 0bff61c Compare April 6, 2020 06:52

mgmarino force-pushed the add-year-for-week-of-year branch 2 times, most recently from 6f41853 to 60d7088 Compare April 6, 2020 14:42

mroeschke reviewed Apr 6, 2020

View reviewed changes

mgmarino added 3 commits April 8, 2020 06:47

Return Int64 dataframe when NaT present

ff7b7e6

Clean up example to use a single Series

330a103

Always return an Int64 data frame

9c30351

mgmarino force-pushed the add-year-for-week-of-year branch from 6064812 to 9c30351 Compare April 8, 2020 04:48

jreback requested changes Apr 8, 2020

View reviewed changes

mgmarino added 2 commits April 8, 2020 18:20

Return UInt32 per calendar ops standard

f40d2f0

Add timeseries documentation

aa6a193

jreback added this to the 1.1 milestone Apr 8, 2020

jreback approved these changes Apr 8, 2020

View reviewed changes

mroeschke approved these changes Apr 8, 2020

View reviewed changes

Merge branch 'master' into add-year-for-week-of-year

b495bfd

mroeschke merged commit 03b510c into pandas-dev:master Apr 12, 2020

mgmarino deleted the add-year-for-week-of-year branch April 13, 2020 18:48

mgmarino added a commit to mgmarino/pandas that referenced this pull request Apr 13, 2020

CLN: Change isocalendar to be a method

614e2ed

For consistency with `Timestamp.isocalendar`, this should rather be a method. Followup of pandas-dev#33220, see the discussions following the merge

mgmarino mentioned this pull request Apr 13, 2020

CLN: Change isocalendar to be a method #33533

Merged

mgmarino added a commit to mgmarino/pandas that referenced this pull request Apr 14, 2020

CLN: Change isocalendar to be a method

b7a89d8

For consistency with `Timestamp.isocalendar`, this should rather be a method. Followup of pandas-dev#33220, see the discussions following the merge

mroeschke pushed a commit that referenced this pull request Apr 14, 2020

CLN: Change isocalendar to be a method (#33533)

041dc44

For consistency with `Timestamp.isocalendar`, this should rather be a method. Followup of #33220, see the discussions following the merge

mgmarino mentioned this pull request Apr 16, 2020

DEPR: Deprecate week, weekofyear in Series.dt,DatetimeIndex #33595

Merged

5 tasks

WillAyd mentioned this pull request Jun 15, 2020

Backport PR #34718 on branch 1.0.x (Removed __div__ impls) #34802

Merged

rth mentioned this pull request Nov 26, 2020

Fix minimal version for pandas ramp-kits/air_passengers#13

Open

rhshadrach mentioned this pull request Jul 9, 2021

BUG: .dt.isocalendar().week returns unexpected dtype (UInt32) #42429

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: Add isocalendar accessor to DatetimeIndex and Series.dt #33220

ENH: Add isocalendar accessor to DatetimeIndex and Series.dt #33220

mgmarino commented Apr 1, 2020 •

edited

Loading

WillAyd commented Apr 1, 2020 •

edited

Loading

mroeschke commented Apr 1, 2020

mgmarino commented Apr 2, 2020

mroeschke commented Apr 2, 2020

mgmarino commented Apr 2, 2020

mroeschke commented Apr 2, 2020

mgmarino commented Apr 2, 2020 •

edited

Loading

mroeschke commented Apr 2, 2020

mgmarino commented Apr 4, 2020

mgmarino commented Apr 4, 2020

mroeschke commented Apr 6, 2020

mgmarino commented Apr 6, 2020

jbrockmendel commented Apr 6, 2020

mroeschke Apr 6, 2020

mgmarino commented Apr 8, 2020

jreback left a comment

jreback Apr 8, 2020

jreback Apr 8, 2020

jreback Apr 8, 2020

mgmarino commented Apr 8, 2020

jreback commented Apr 8, 2020

jreback commented Apr 10, 2020

mroeschke commented Apr 12, 2020

jorisvandenbossche commented Apr 13, 2020

mroeschke commented Apr 13, 2020

mgmarino commented Apr 13, 2020

ENH: Add isocalendar accessor to DatetimeIndex and Series.dt #33220

ENH: Add isocalendar accessor to DatetimeIndex and Series.dt #33220

Conversation

mgmarino commented Apr 1, 2020 • edited Loading

WillAyd commented Apr 1, 2020 • edited Loading

mroeschke commented Apr 1, 2020

mgmarino commented Apr 2, 2020

mroeschke commented Apr 2, 2020

mgmarino commented Apr 2, 2020

mroeschke commented Apr 2, 2020

mgmarino commented Apr 2, 2020 • edited Loading

mroeschke commented Apr 2, 2020

mgmarino commented Apr 4, 2020

mgmarino commented Apr 4, 2020

mroeschke commented Apr 6, 2020

mgmarino commented Apr 6, 2020

jbrockmendel commented Apr 6, 2020

mroeschke Apr 6, 2020

Choose a reason for hiding this comment

mgmarino commented Apr 8, 2020

jreback left a comment

Choose a reason for hiding this comment

jreback Apr 8, 2020

Choose a reason for hiding this comment

jreback Apr 8, 2020

Choose a reason for hiding this comment

jreback Apr 8, 2020

Choose a reason for hiding this comment

mgmarino commented Apr 8, 2020

jreback commented Apr 8, 2020

jreback commented Apr 10, 2020

mroeschke commented Apr 12, 2020

jorisvandenbossche commented Apr 13, 2020

mroeschke commented Apr 13, 2020

mgmarino commented Apr 13, 2020

mgmarino commented Apr 1, 2020 •

edited

Loading

WillAyd commented Apr 1, 2020 •

edited

Loading

mgmarino commented Apr 2, 2020 •

edited

Loading