Skip to content

[FLINK-40187][python] Introduce DataType class in DataFrame API - #28843

Merged
dianfu merged 2 commits into
apache:masterfrom
auroflow:auroflow/dataframe-types
Aug 3, 2026
Merged

[FLINK-40187][python] Introduce DataType class in DataFrame API#28843
dianfu merged 2 commits into
apache:masterfrom
auroflow:auroflow/dataframe-types

Conversation

@auroflow

@auroflow auroflow commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

What is the purpose of the change

This pull request completes the DataType API for the PyFlink DataFrame API. It provides the scalar, temporal, and composite data types required by subsequent DataFrame operations.

Brief change log

  • Add scalar, temporal, and composite DataType factory methods.
  • Add nullability modifiers and a readable representation for DataType.
  • Add internal helpers for inferring data types from Python type hints and parsing SQL type strings.
  • Preserve timestamp precision and support NULL when converting Java data types to Python data types.
  • Extend the DataFrame API reference documentation.
  • Add focused tests for factories, nullability, nested types, type inference, SQL parsing, equality, hashing, and representation.

Verifying this change

This change added tests and can be verified as follows:

  • Ran the complete DataFrame test suite:
    • .venv/bin/python -m pytest -q flink-python/pyflink/dataframe/tests
    • Result: 94 passed
  • Ran flake8 checks for all changed Python files.
  • Compiled all changed Python files with py_compile.
  • Ran mvn -T 12 spotless:check through the pre-push hook.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): yes
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? docs and Python API docstrings
Was generative AI tooling used to co-author this PR?
  • Yes (OpenAI Codex)

Generated-by: OpenAI Codex (GPT-5.6 Sol)

@flinkbot

flinkbot commented Jul 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@auroflow
auroflow force-pushed the auroflow/dataframe-types branch from f840c98 to ff4525b Compare July 30, 2026 09:22
@auroflow
auroflow force-pushed the auroflow/dataframe-types branch from ff4525b to d8682e9 Compare July 30, 2026 09:55
@auroflow
auroflow marked this pull request as ready for review July 30, 2026 11:29
dianfu
dianfu reviewed Jul 31, 2026
View reviewed changes
Comment thread flink-python/docs/reference/pyflink.dataframe/datatype.rst Outdated

DataType
DataType.not_null
DataType.nullable

@dianfu dianfu Jul 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about moving these two APIs to the end of this list?

@auroflow auroflow Aug 1, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved these null markers to the end of the list.

Comment thread flink-python/pyflink/dataframe/datatype.py
"""
Create a list type.

:param dtype: Type of each list element.

@dianfu dianfu Jul 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be great to add an example.

@auroflow auroflow Aug 1, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added.

Comment thread flink-python/pyflink/dataframe/datatype.py
Create a map type.

:param key_type: Type of each map key.
:param value_type: Type of each map value.

@dianfu dianfu Jul 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto, it would be great to add an example.

@auroflow auroflow Aug 1, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added.

Comment thread flink-python/pyflink/dataframe/datatype.py
>>> import pyflink.dataframe as pf
>>> person_type = pf.DataType.struct({
... "name": pf.DataType.string(),
... "age": pf.DataType.int32(),

@dianfu dianfu Jul 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also add an example when the type of fields is an list.

@auroflow auroflow Aug 1, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added.

Comment thread flink-python/pyflink/dataframe/datatype.py

@classmethod
@PublicEvolving()
def int8(cls) -> "DataType":

@dianfu dianfu Jul 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pf.lit(1, DataType.int8()), int16(), and float32() raise Java ValidationExceptions because Py4J sends Integer/Double rather than Byte/Short/Float. Date/time/timestamp, list, and struct literals fail similarly. Extend the literal conversion path for these types and add end-to-end factory/literal tests.

@auroflow auroflow Aug 1, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! The root cause for this is an existing flaw in the Table API's lit method, which fails to adapt some types for Py4J. I think it's worth a separate PR, where we fix it cleanly in the Table API layer.

@dianfu dianfu Aug 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. Could you create a JIRA ticket?

@auroflow auroflow Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created at FLINK-40300. I'll be working on a PR soon.

@auroflow
[FLINK-40187][python] Address review comments on DataType docstrings
7d1396d 
Move DataType.not_null and DataType.nullable to the end of the
autosummary list in the RST reference, and add usage examples to the
DataType.list, DataType.map, and DataType.struct docstrings (including
the list-of-tuples form for struct).

Generated-by: OpenCode (GLM-5.2-FP8)
@auroflow
auroflow force-pushed the auroflow/dataframe-types branch from 9af8b75 to 7d1396d Compare August 1, 2026 06:40
@dianfu
dianfu merged commit 0910fc9 into apache:master Aug 3, 2026
@auroflow
auroflow deleted the auroflow/dataframe-types branch August 3, 2026 03:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dianfu dianfu dianfu approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@auroflow @flinkbot @dianfu