PERF: Faster transposition of frames with masked arrays #52836

topper-123 · 2023-04-21T18:53:59Z

Faster transpose of dataframes with homogenous masked arrays. This also helps when doing reductions with axis=1 as those currently transpose data before doing reductions.

Performance example:

In [1]:         import pandas as pd, numpy as np
   ...:         values = np.random.randn(100000, 4)
   ...:         values = values.astype(int)
   ...:         df = pd.DataFrame(values).astype("Int64")
>>> %timeit df.transpose()
1.91 s ± 3.08 ms per loop  # main
563 ms ± 2.48 ms per loop  # this PR

Running asv continuous -f 1.1 upstream/main HEAD -b reshape.ReshapeMaskedArrayDtype.time_transpose:

       before           after         ratio
     [c3f0aac1]       [43162428]
     <master>         <cln_managers>
-     13.5±0.06ms      1.82±0.01ms     0.14  reshape.ReshapeMaskedArrayDtype.time_transpose('Float64')
-     13.8±0.03ms      1.83±0.01ms     0.13  reshape.ReshapeMaskedArrayDtype.time_transpose('Int64')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.

There may be possible to improve performance when frames have a common masked dtype, I intend to look into that in a followup.

jorisvandenbossche · 2023-04-21T20:34:16Z

@topper-123 have you seen #52083 and #52689?
I think it is generally the same idea (directly accessing _data and _mask and creating the transposed versions of those separately, and then faster reconstructing the MaskedArrays from that), but a bit differently organized.

rhshadrach · 2023-04-25T22:08:02Z

Using the timings in the OP, I'm seeing

625 ms ± 8.04 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  # <-- This PR
733 ms ± 3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) # <-- #52689

I haven't yet looked into the implementation to see where this difference is coming from. I'm happy to go with this implementation, would like to see if we can get this in 2.0.2.

topper-123 · 2023-04-26T07:41:43Z

I've merged the latest changes to the main branch into this in case this is he one we go with.

rhshadrach

lgtm, cc @jorisvandenbossche @jbrockmendel

jbrockmendel · 2023-05-06T20:17:24Z

pandas/core/arrays/masked.py

@@ -1414,3 +1414,32 @@ def _groupby_op(
        # res_values should already have the correct dtype, we just need to
        #  wrap in a MaskedArray
        return self._maybe_mask_result(res_values, result_mask)
+
+
+def transpose_homogenous_masked_arrays(masked_arrays):


annotation?

jbrockmendel · 2023-05-06T20:18:01Z

pandas/core/arrays/masked.py

+    transposed_values = np.empty(transposed_shape, dtype=values[0].dtype)
+    for i, val in enumerate(values):
+        transposed_values[i, :] = val
+    transposed_values = transposed_values.copy(order="F")


this is faster than np.concatenate?

It's about the same. I'll change to use np.concatenate, it's fewer lines.

jbrockmendel · 2023-05-06T20:19:56Z

pandas/core/frame.py

+                if isinstance(self._mgr, ArrayManager):
+                    masked_arrays = self._mgr.arrays
+                else:
+                    masked_arrays = [blk.values for blk in self._mgr.blocks]


might be safer to use iter_column_arrays? this might mess up if blocks get shuffled in a weird order

I've changed it.

topper-123 · 2023-05-14T17:41:31Z

ping...

rhshadrach · 2023-05-17T21:17:27Z

@topper-123 - have a conflict. @jbrockmendel - friendly ping.

* Improve performance when selecting rows and columns * Update doc/source/whatsnew/v2.1.0.rst Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Update indexing.py * Update v2.1.0.rst --------- Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com>

…3088) * PERF: Improve performance when accessing GroupBy.groups * Update v2.1.0.rst * Fix

* Improve performance when selecting rows and columns * Update doc/source/whatsnew/v2.1.0.rst Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> * Update indexing.py * Update v2.1.0.rst --------- Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com>

…3088) * PERF: Improve performance when accessing GroupBy.groups * Update v2.1.0.rst * Fix

topper-123 · 2023-06-03T10:25:27Z

Ping.

github-actions · 2023-07-12T00:06:25Z

This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this.

topper-123 · 2023-07-13T11:45:58Z

The PR has gone quite stale, I think it's best to close this PR, unless it can be approved for merging.

cc @jbrockmendel , @rhshadrach .

rhshadrach

lgtm; two minor requests and a conflict and I think we're good here.

pandas/core/arrays/masked.py

pandas/core/frame.py

topper-123 · 2023-07-15T17:40:47Z

I've updated.

rhshadrach

lgtm

rhshadrach · 2023-07-16T19:53:55Z

Thanks @topper-123

topper-123 · 2023-07-16T20:43:26Z

Thanks.

In total, after the other related PRs have been merged, we have:

n [1]:         import pandas as pd, numpy as np
   ...:         values = np.random.randn(100000, 4)
   ...:         values = values.astype(int)
   ...:         df = pd.DataFrame(values).astype("Int64")
>>> %timeit df.transpose()
1.91 s ± 3.08 ms per loop  # main at start
563 ms ± 2.48 ms per loop  # this PR, originally
375 ms ± 5.47 ms per loop  # this PR, now

An improvement, but very slow still...

rhshadrach added Reshaping Concat, Merge/Join, Stack/Unstack, Explode NA - MaskedArrays Related to pd.NA and nullable extension arrays Performance Memory or execution speed performance labels Apr 22, 2023

rhshadrach approved these changes Apr 28, 2023

View reviewed changes

This was referenced Apr 29, 2023

PERF: faster access to the dtype for masked numeric arrays #52998

Merged

PERF: add ._simple_new method to masked arrays #53013

Merged

ENH: better dtype inference when doing DataFrame reductions #52788

Merged

jbrockmendel reviewed May 6, 2023

View reviewed changes

topper-123 force-pushed the faster_masked_transpose branch 3 times, most recently from 0a878f4 to f36143c Compare May 9, 2023 06:29

topper-123 and others added 12 commits May 27, 2023 08:35

PERF: Faster transposition of masked arrays

f250ea4

simplify ASVs

deaa6bb

typing failed, less typing

3e56638

pre-commit fixes

95230dd

update

3847c48

PERF: Improve performance when accessing GroupBy.groups (pandas-dev#5…

cb7fbe9

…3088) * PERF: Improve performance when accessing GroupBy.groups * Update v2.1.0.rst * Fix

changes according to comments

e065f4d

fix precommit

b664503

PERF: Faster transposition of masked arrays

29c03c6

update

2af2fc3

phofl and others added 2 commits May 27, 2023 08:37

PERF: Improve performance when accessing GroupBy.groups (pandas-dev#5…

22f5238

…3088) * PERF: Improve performance when accessing GroupBy.groups * Update v2.1.0.rst * Fix

fix precommit

1852bb2

topper-123 force-pushed the faster_masked_transpose branch from 7188f20 to 1852bb2 Compare May 27, 2023 07:39

topper-123 added 2 commits May 27, 2023 08:57

fix pre-commit

502bcee

Merge branch 'main' into faster_masked_transpose

bf212dd

Merge branch 'master' into faster_masked_transpose

047564d

github-actions bot added the Stale label Jul 12, 2023

rhshadrach requested changes Jul 15, 2023

View reviewed changes

pandas/core/arrays/masked.py Outdated Show resolved Hide resolved

pandas/core/frame.py Outdated Show resolved Hide resolved

topper-123 added 4 commits July 15, 2023 09:11

Merge branch 'master' into faster_masked_transpose

c61ba7d

update according to comments

07663af

update according to comments II

094ee0a

fix pre-commit

7234b03

rhshadrach approved these changes Jul 16, 2023

View reviewed changes

rhshadrach merged commit f1211e7 into pandas-dev:main Jul 16, 2023
36 checks passed

rhshadrach added this to the 2.1 milestone Jul 16, 2023

topper-123 deleted the faster_masked_transpose branch July 16, 2023 20:26

lukemanley mentioned this pull request Jul 22, 2023

PERF: DataFrame.transpose for pyarrow-backed #54224

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PERF: Faster transposition of frames with masked arrays #52836

PERF: Faster transposition of frames with masked arrays #52836

topper-123 commented Apr 21, 2023 •

edited

jorisvandenbossche commented Apr 21, 2023

rhshadrach commented Apr 25, 2023

topper-123 commented Apr 26, 2023

rhshadrach left a comment

jbrockmendel May 6, 2023

topper-123 May 7, 2023

jbrockmendel May 6, 2023

topper-123 May 7, 2023

jbrockmendel May 6, 2023

topper-123 May 7, 2023

topper-123 commented May 14, 2023

rhshadrach commented May 17, 2023

topper-123 commented Jun 3, 2023

github-actions bot commented Jul 12, 2023

topper-123 commented Jul 13, 2023

rhshadrach left a comment •

edited

topper-123 commented Jul 15, 2023

rhshadrach left a comment

rhshadrach commented Jul 16, 2023

topper-123 commented Jul 16, 2023 •

edited

PERF: Faster transposition of frames with masked arrays #52836

PERF: Faster transposition of frames with masked arrays #52836

Conversation

topper-123 commented Apr 21, 2023 • edited

jorisvandenbossche commented Apr 21, 2023

rhshadrach commented Apr 25, 2023

topper-123 commented Apr 26, 2023

rhshadrach left a comment

Choose a reason for hiding this comment

jbrockmendel May 6, 2023

Choose a reason for hiding this comment

topper-123 May 7, 2023

Choose a reason for hiding this comment

jbrockmendel May 6, 2023

Choose a reason for hiding this comment

topper-123 May 7, 2023

Choose a reason for hiding this comment

jbrockmendel May 6, 2023

Choose a reason for hiding this comment

topper-123 May 7, 2023

Choose a reason for hiding this comment

topper-123 commented May 14, 2023

rhshadrach commented May 17, 2023

topper-123 commented Jun 3, 2023

github-actions bot commented Jul 12, 2023

topper-123 commented Jul 13, 2023

rhshadrach left a comment • edited

Choose a reason for hiding this comment

topper-123 commented Jul 15, 2023

rhshadrach left a comment

Choose a reason for hiding this comment

rhshadrach commented Jul 16, 2023

topper-123 commented Jul 16, 2023 • edited

topper-123 commented Apr 21, 2023 •

edited

rhshadrach left a comment •

edited

topper-123 commented Jul 16, 2023 •

edited