Skip to content

[FLINK-40236][python] Fix _infer_type inferring ARRAY<NULL> for a list with a leading None#28819

Open
nikolauspschuetz wants to merge 2 commits into
apache:masterfrom
nikolauspschuetz:fix-flink-40236-infer-array-leading-none
Open

[FLINK-40236][python] Fix _infer_type inferring ARRAY<NULL> for a list with a leading None#28819
nikolauspschuetz wants to merge 2 commits into
apache:masterfrom
nikolauspschuetz:fix-flink-40236-infer-array-leading-none

Conversation

@nikolauspschuetz

@nikolauspschuetz nikolauspschuetz commented Jul 25, 2026

Copy link
Copy Markdown

What is the purpose of the change

_infer_type in flink-python/pyflink/table/types.py inferred a list's element type from obj[0] instead of the first non-None element it scans for. When the first element is None, the array element type collapsed to NULL, making schema inference order-dependent:

_infer_type([1, None]) -> ArrayType(BigIntType)   # correct
_infer_type([None, 1]) -> ArrayType(NullType)     # wrong; should be BigIntType

The dict branch just above already handles this correctly (it infers from the found non-None value). from_elements([Row(c=[None, 1])]) without an explicit schema infers ARRAY<NULL>, which then typically errors downstream.

Brief change log

  • Infer the array element type from the scanned non-None element v rather than obj[0].

Verifying this change

This change added a test and can be verified as follows:

  • Added TypesTests.test_infer_array_type_with_leading_none asserting _infer_type([None, 1]) yields ArrayType(BigIntType).
  • Verified against the released apache-flink wheel: the array element type is NullType before the fix and BigIntType after (the all-None case still yields ARRAY<NULL>).

The two commits demonstrate this: the first adds the test only (CI red — test_infer_array_type_with_leading_none fails with AssertionError: False is not true), the second applies the fix (CI green). Runs on my fork:

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no

Was generative AI tooling used to co-author this PR?
  • Yes (Claude Code)

Generated-by: Claude Code (Opus 4.8)

@flinkbot

flinkbot commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

Demonstrates the bug: _infer_type([None, 1]) currently yields
ArrayType(NullType) because the element type is inferred from obj[0]
rather than the first non-None element. Fix follows in the next commit.
@nikolauspschuetz
nikolauspschuetz force-pushed the fix-flink-40236-infer-array-leading-none branch from e1903b7 to 10917a2 Compare July 25, 2026 04:35
…ement

_infer_type inferred a list's element type from obj[0] rather than the
first non-None element the loop scans for, so a leading None collapsed
the array element type to NULL. Infer from the scanned element v
instead, matching the dict branch above.

This makes the test added in the previous commit pass.
@nikolauspschuetz
nikolauspschuetz marked this pull request as ready for review July 25, 2026 17:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants