PERF: RangeIndex.format performance #35712

topper-123 · 2020-08-13T18:39:42Z

#35440 dropped RangeIndex._format_with_header, which was functionally not needed, but needed to avoid creating an internal ndarray.

This rectifies that + gives some perf. improvements.

>>> idx = pd.RangeIndex(1_000_000)
>>> %timeit idx.format()
4.6 s ± 102 ms per loop  # pandas v1.1.0
1.67 s ± 19.6 ms per loop  # master
595 ms ± 2.35 ms per loop  # this PR

Also, now the _data attribute isn't called, so this PR gives a perf. & memory improvement in some use cases compared to master:

>>> idx = pd.RangeIndex(1_000_000)
>>> idx.format()
>>> "_data" in idx._cache
False # pandas v.1.1.0
True  # master
False  # this PR

arw2019

lgtm

Do we have benchmarks for this? If not do we want to add them?

jreback · 2020-08-14T13:42:39Z

pandas/core/indexes/range.py

@@ -187,6 +187,11 @@ def _format_data(self, name=None):
        # we are formatting thru the attributes
        return None

+    def _format_with_header(self, header, na_rep="NaN") -> List[str]:


can you type

jreback · 2020-08-19T22:54:08Z

looks good. this is worth a perf mention in 1.2 whatsnew. ping on green.

topper-123 · 2020-08-20T22:04:04Z

The TravisCI issue is just a timeout issue, but In CI / Web and docs, I've got a problem under Check ipython directive errors that I'm not able to fix:

Run ! grep -B1 "^<<<-------------------------------------------------------------------------$" sphinx.log
[0 rows x 1 columns]
<<<-------------------------------------------------------------------------
##[error]Process completed with exit code 1.

The source code for this in .github/workflows/ci.yml says:

    # This can be removed when the ipython directive fails when there are errors,
    # including the `tee sphinx.log` in te previous step (https://github.com/ipython/ipython/issues/11547)
    - name: Check ipython directive errors
      run: "! grep -B1 \"^<<<-------------------------------------------------------------------------$\" sphinx.log"

I don't understand what this check is for or why my PR fails here. @TomAugspurger, you posted that mentioned ipython issue, do you mind looking at why my PR fails on this?

TomAugspurger

There's an exception in the whatsnew/0.25.0.rst file: https://github.com/pandas-dev/pandas/pull/35712/checks?check_run_id=1010393272#step:6:2305

pandas/core/indexes/range.py

topper-123 · 2020-08-21T20:15:33Z

Updated.

jreback · 2020-08-21T21:59:00Z

lgtm. I think this is ok for 1.1.2 as a perf regresion, can you move the note? ping on green.

topper-123 · 2020-08-22T11:00:03Z

Updated & green.

doc/source/whatsnew/v1.1.2.rst

simonjayhawkins · 2020-08-24T12:06:27Z

pandas/core/indexes/range.py

@@ -187,6 +187,15 @@ def _format_data(self, name=None):
        # we are formatting thru the attributes
        return None

+    def _format_with_header(self, header: List[str], na_rep: str = "NaN") -> List[str]:
+        if len(self._range) == 0:
+            return []


should this be return header?

Yes.

I added tests for this, and found that DateTimeIndex([]).format(name=True) returns ["None"], where it should return [""]. Same for PeriodIndex. I fixed that bug.

simonjayhawkins · 2020-08-25T09:51:14Z

@topper-123 merge conflicts in whatsnew

simonjayhawkins

Thanks @topper-123 lgtm pending green

jreback

minor comment, if you can address and merge on green.

jreback · 2020-08-25T12:54:32Z

pandas/core/indexes/range.py

@@ -187,6 +187,15 @@ def _format_data(self, name=None):
        # we are formatting thru the attributes
        return None

+    def _format_with_header(self, header: List[str], na_rep: str = "NaN") -> List[str]:
+        if len(self._range) == 0:


if not len(self._range)

into RangeIndex.format

topper-123 · 2020-08-25T16:13:06Z

That merge brancg master has caused some strange error. anyone know how to fix it?

… RangeIndex.format

topper-123 · 2020-08-26T09:57:22Z

Thanks, @simonjayhawkins, were there any git oneliners you used for this?

simonjayhawkins · 2020-08-26T10:40:52Z

Thanks, @simonjayhawkins, were there any git oneliners you used for this?

I don't profess to know what was going on. resetting HEAD, merging master and then pulling from the branch seemed to fix except for the duplicate release note.

simonjayhawkins · 2020-08-26T11:12:41Z

minor comment, if you can address and merge on green.

comment addressed

simonjayhawkins · 2020-08-26T11:13:08Z

Thanks @topper-123

simonjayhawkins · 2020-08-26T11:14:20Z

@meeseeksdev backport 1.1.x

lumberbot-app · 2020-08-26T11:14:45Z

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

Checkout backport branch and update it.

$ git checkout 1.1.x
$ git pull

Cherry pick the first parent branch of the this PR on top of the older branch:

$ git cherry-pick -m1 b7a31ebeb873f37939838b3b788dab0e4c41b8de

You will likely have some merge/cherry-pick conflict here, fix them and commit:

$ git commit -am 'Backport PR #35712: PERF: RangeIndex.format performance'

Push to a named branch :

git push YOURFORK 1.1.x:auto-backport-of-pr-35712-on-1.1.x

Create a PR against branch 1.1.x, I would have named this PR:

"Backport PR #35712 on branch 1.1.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to suggest an improvement.

Co-authored-by: Terji Petersen <contribute@tensortable.com>

arw2019 reviewed Aug 14, 2020

View reviewed changes

jreback added Output-Formatting __repr__ of pandas objects, to_string Performance Memory or execution speed performance labels Aug 14, 2020

jreback requested changes Aug 14, 2020

View reviewed changes

topper-123 force-pushed the RangeIndex.format branch 3 times, most recently from bbb33d5 to 5ff29f3 Compare August 19, 2020 17:43

jreback added this to the 1.2 milestone Aug 19, 2020

TomAugspurger reviewed Aug 21, 2020

View reviewed changes

pandas/core/indexes/range.py Show resolved Hide resolved

jreback modified the milestones: 1.2, 1.1.2 Aug 21, 2020

topper-123 force-pushed the RangeIndex.format branch from c08d426 to 667b501 Compare August 22, 2020 10:00

topper-123 changed the title ~~PERF: RangeIndex.format perf regression~~ PERF: RangeIndex.format performance Aug 22, 2020

topper-123 force-pushed the RangeIndex.format branch from 667b501 to 224b46e Compare August 22, 2020 10:02

simonjayhawkins reviewed Aug 24, 2020

View reviewed changes

topper-123 added 8 commits August 24, 2020 17:44

PERF: RangeIndex.format perf regression

54dcde1

fix errors

e36403d

TYP: add types to params

fef0b2c

add more types

5998b5e

fix indention in directive

6ffdb82

fix zero length RangeIndex + add tests

f5759e0

add whatsnew

e1476a9

fix name=True bug

041d4df

topper-123 force-pushed the RangeIndex.format branch from 224b46e to 041d4df Compare August 24, 2020 17:14

simonjayhawkins approved these changes Aug 25, 2020

View reviewed changes

Merge branch 'master' into RangeIndex.format

f18c7a4

jreback approved these changes Aug 25, 2020

View reviewed changes

topper-123 added 10 commits August 25, 2020 16:44

PERF: RangeIndex.format perf regression

b402d47

fix errors

af1e400

TYP: add types to params

d6b3b35

add more types

d129776

fix indention in directive

4c697af

fix zero length RangeIndex + add tests

b3f47c9

add whatsnew

1e813db

fix name=True bug

db932e5

small change

d88202a

Merge branch 'RangeIndex.format' of https://github.com/topper-123/pandas

e1ba9c1

into RangeIndex.format

simonjayhawkins added 3 commits August 26, 2020 10:39

Merge remote-tracking branch 'upstream/master' into RangeIndex.format

bdeb85a

Merge branch 'RangeIndex.format' of github.com:topper-123/pandas into…

5db5d9b

… RangeIndex.format

remove dup whatsnew entry

b90c8fe

simonjayhawkins merged commit b7a31eb into pandas-dev:master Aug 26, 2020

lumberbot-app bot added the Still Needs Manual Backport label Aug 26, 2020

simonjayhawkins pushed a commit to simonjayhawkins/pandas that referenced this pull request Aug 26, 2020

Backport PR pandas-dev#35712: PERF: RangeIndex.format performance

731b38e

simonjayhawkins mentioned this pull request Aug 26, 2020

Backport PR #35712: PERF: RangeIndex.format performance #35904

Merged

simonjayhawkins removed the Still Needs Manual Backport label Aug 26, 2020

topper-123 deleted the RangeIndex.format branch August 26, 2020 13:29

simonjayhawkins added a commit that referenced this pull request Aug 26, 2020

Backport PR #35712: PERF: RangeIndex.format performance (#35904)

f86b129

Co-authored-by: Terji Petersen <contribute@tensortable.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PERF: RangeIndex.format performance #35712

PERF: RangeIndex.format performance #35712

topper-123 commented Aug 13, 2020 •

edited

Loading

arw2019 left a comment

jreback Aug 14, 2020

jreback commented Aug 19, 2020

topper-123 commented Aug 20, 2020 •

edited

Loading

TomAugspurger left a comment

topper-123 commented Aug 21, 2020

jreback commented Aug 21, 2020

topper-123 commented Aug 22, 2020

simonjayhawkins Aug 24, 2020

topper-123 Aug 24, 2020

simonjayhawkins commented Aug 25, 2020

simonjayhawkins left a comment

jreback left a comment

jreback Aug 25, 2020

topper-123 commented Aug 25, 2020

topper-123 commented Aug 26, 2020

simonjayhawkins commented Aug 26, 2020

simonjayhawkins commented Aug 26, 2020

simonjayhawkins commented Aug 26, 2020

simonjayhawkins commented Aug 26, 2020

lumberbot-app bot commented Aug 26, 2020

PERF: RangeIndex.format performance #35712

PERF: RangeIndex.format performance #35712

Conversation

topper-123 commented Aug 13, 2020 • edited Loading

arw2019 left a comment

Choose a reason for hiding this comment

jreback Aug 14, 2020

Choose a reason for hiding this comment

jreback commented Aug 19, 2020

topper-123 commented Aug 20, 2020 • edited Loading

TomAugspurger left a comment

Choose a reason for hiding this comment

topper-123 commented Aug 21, 2020

jreback commented Aug 21, 2020

topper-123 commented Aug 22, 2020

simonjayhawkins Aug 24, 2020

Choose a reason for hiding this comment

topper-123 Aug 24, 2020

Choose a reason for hiding this comment

simonjayhawkins commented Aug 25, 2020

simonjayhawkins left a comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

jreback Aug 25, 2020

Choose a reason for hiding this comment

topper-123 commented Aug 25, 2020

topper-123 commented Aug 26, 2020

simonjayhawkins commented Aug 26, 2020

simonjayhawkins commented Aug 26, 2020

simonjayhawkins commented Aug 26, 2020

simonjayhawkins commented Aug 26, 2020

lumberbot-app bot commented Aug 26, 2020

topper-123 commented Aug 13, 2020 •

edited

Loading

topper-123 commented Aug 20, 2020 •

edited

Loading