Skip to content

[FLINK-40423][python] Add Top-N support to the DataFrame API - #29105

Open
Timm0 wants to merge 1 commit into
apache:masterfrom
Timm0:FLINK-40423
Open

[FLINK-40423][python] Add Top-N support to the DataFrame API#29105
Timm0 wants to merge 1 commit into
apache:masterfrom
Timm0:FLINK-40423

Conversation

@Timm0

@Timm0 Timm0 commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

What is the purpose of the change

This change adds DataFrame.top_n(...) for global and per-group Top-N, and refactors the existing drop_duplicates onto the same shared SQL builder so both produce the same plan (StreamPhysicalRank, or StreamPhysicalDeduplicate for n == 1 on a time attribute).

Brief change log

  • Add DataFrame.top_n
  • Generalize _build_deduplication_query into the shared _build_rank_sql
  • Route drop_duplicates through _build_rank_sql
  • Add _normalize_descending helper
  • Move _materialize test helper to PyFlinkStreamDataFrameTestCase
  • Add DataFrameTopNTests + DataFrameTopNITTests
  • List DataFrame.top_n in the API reference

Verifying this change

  • Added unit tests: DataFrameTopNTests
  • Added integration tests: DataFrameTopNITTests

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): yes
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? docs flink-python/docs/reference/pyflink.dataframe/dataframe.rst + Python docstrings

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Opus 4.8 (1M context)

- Add `DataFrame.top_n`
- Generalize `_build_deduplication_query` into the shared `_build_rank_sql`
- Route `drop_duplicates` through `_build_rank_sql`
- Add `_normalize_descending` helper
- Move `_materialize` test helper to `PyFlinkStreamDataFrameTestCase`
- Add `DataFrameTopNTests` + `DataFrameTopNITTests`
- List `DataFrame.top_n` in the API reference
@Timm0
Timm0 marked this pull request as ready for review September 4, 2026 14:30
@flinkbot

flinkbot commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants