Skip to content

[FLINK-40421][python] Add sorting APIs to DataFrame API - #29078

Open
Milesian111 wants to merge 1 commit into
apache:masterfrom
Milesian111:pull/40421
Open

[FLINK-40421][python] Add sorting APIs to DataFrame API#29078
Milesian111 wants to merge 1 commit into
apache:masterfrom
Milesian111:pull/40421

Conversation

@Milesian111

Copy link
Copy Markdown
Contributor

What is the purpose of the change

This pull request adds sorting support to the PyFlink DataFrame API. It introduces a lazy DataFrame.sort() transformation backed by the existing Table API order_by() operation, allowing users to sort by one or more columns or expressions without triggering execution.

Brief change log

  • Added DataFrame.sort() with support for single-column and multi-column sorting.
  • Added scalar and per-column descending options.
  • Added scalar and per-column nulls_first options.
  • Added validation to reject expressions that already contain .asc or .desc.
  • Kept sorting lazy and ensured the original DataFrame remains unchanged.
  • Added public API documentation.

Verifying this change

This change added focused tests covering:

  • Single-column and multi-column sorting.
  • Expression-based sorting.
  • Scalar and per-column sort directions.
  • Explicit NULL ordering.
  • Invalid inputs and mismatched option lengths.
  • Lazy plan construction and DataFrame immutability.
  • Batch execution with deterministic ordering and NULL handling.

The focused sorting tests and complete DataFrame batch integration tests pass locally. The full DataFrame test module has one unrelated Windows-only failure involving pre-epoch timestamp conversion in time.mktime.

Does this pull request potentially affect one of the following parts:

  • Dependencies: no
  • The public API: yes
  • The serializers: no
  • The runtime per-record code paths: no
  • Anything that affects deployment or recovery: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? docs

Was generative AI tooling used to co-author this PR?
  • Yes

Generated-by: Codex GPT-5

@flinkbot

flinkbot commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

Add lazy DataFrame.sort backed by Table.order_by with per-key direction and explicit null ordering. Reject ordered expressions and cover validation, planning, batch behavior, and mypy-compatible typing with focused tests.

Generated-by: Codex GPT-5
@Milesian111

Copy link
Copy Markdown
Contributor Author

@flinkbot run azure

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants