Skip to content

[SPARK-58971][PS] Add pct, na_option, and axis parameters to Series.rank and DataFrame.rank - #58249

Closed
jzhan-2026 wants to merge 2 commits into
apache:masterfrom
jzhan-2026:master
Closed

[SPARK-58971][PS] Add pct, na_option, and axis parameters to Series.rank and DataFrame.rank#58249
jzhan-2026 wants to merge 2 commits into
apache:masterfrom
jzhan-2026:master

Conversation

@jzhan-2026

Copy link
Copy Markdown

What changes were proposed in this pull request?

Add na_option, pct, and axis parameters to Series.rank(), and na_option and pct to
DataFrame.rank(), to match the pandas API.

  • na_option ('keep'/'top'/'bottom'): controls how NaN values are ranked
  • pct: expresses ranks as percentile fractions in (0, 1]
  • axis on Series.rank(): accepts 0/'index' only, added for API compatibility

Why are the changes needed?

The pandas-on-Spark implementations were missing these parameters, making it harder to migrate
pandas code to Spark without modification. DataFrame.rank was tracked in SPARK-46167; this PR
also covers Series.rank.

Does this PR introduce any user-facing change?

Yes. Two new keyword parameters on Series.rank() and DataFrame.rank(). Defaults are
unchanged so existing code is unaffected.

>>> ps.Series([1, float('nan'), 2]).rank(na_option='top')
0    2.0
1    1.0
2    3.0


>>> ps.Series([1, 2, 2, 3]).rank(pct=True)
0    0.25
1    0.625
2    0.625
3    1.0

How was this patch tested?

New test method test_rank_pct_na_option in test_compute.py and new cases in test_stat.py
covering all methods, both axes, combined parameters, and invalid input errors.

Was this patch authored or co-authored using generative AI tooling?

This patch was co-authored with AI assistance (Claude Sonnet 4.6).

@jzhan-2026 jzhan-2026 changed the title [SPARK-46167][PS] Add pct, na_option, and axis parameters to Series.rank and DataFrame.rank [SPARK-58971][PS] Add pct, na_option, and axis parameters to Series.rank and DataFrame.rank Aug 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant