Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PERF: use the libjoin fast paths in a few more cases #56841

Merged
merged 2 commits into from
Jan 12, 2024

Conversation

lukemanley
Copy link
Member

Additional cases where the fast path can be used:

  • left join with many:1 (right unique)
  • right join with 1:many (left unique)

These are covered by existing tests.

import pandas as pd
import numpy as np

left = pd.Index(np.arange(100_000)).repeat(10)
right = pd.Index(np.arange(100_000))

%timeit left.join(right, how="left")

# 22.7 ms ± 7.84 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
# 3.8 ms ± 60.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
import pandas as pd
import numpy as np

data = [f"i_{i:06}" for i in range(100_000)]
left = pd.Index(data, "string[pyarrow_numpy]")
right = pd.Index(data, "string[pyarrow_numpy]").repeat(10)

%timeit left.join(right, how="right")

# 393 ms ± 4.64 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# 150 ms ± 3.98 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

@lukemanley lukemanley added Performance Memory or execution speed performance Reshaping Concat, Merge/Join, Stack/Unstack, Explode labels Jan 12, 2024
@lukemanley lukemanley added this to the 3.0 milestone Jan 12, 2024
@mroeschke mroeschke merged commit f4f006c into pandas-dev:main Jan 12, 2024
50 checks passed
@mroeschke
Copy link
Member

Nice finds keep them coming!

pmhatre1 pushed a commit to pmhatre1/pandas-pmhatre1 that referenced this pull request May 7, 2024
* use libjoin.left_join_indexer_unique in more cases

* whatsnew
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Performance Memory or execution speed performance Reshaping Concat, Merge/Join, Stack/Unstack, Explode
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants