Skip to content

[FLINK-40203][runtime] Add an explicit option for Flink SQL-compatible YAML Transform expression semantics#4484

Open
haruki-830 wants to merge 1 commit into
apache:masterfrom
haruki-830:FLINK-40203
Open

[FLINK-40203][runtime] Add an explicit option for Flink SQL-compatible YAML Transform expression semantics#4484
haruki-830 wants to merge 1 commit into
apache:masterfrom
haruki-830:FLINK-40203

Conversation

@haruki-830

Copy link
Copy Markdown
Contributor

Summary

This commit adds an explicit expression semantics option to YAML Transform. The default LEGACY mode preserves existing pipeline behavior, while the opt-in FLINK_SQL mode aligns supported predicates with Flink SQL semantics.

Key Changes

  1. Configurable Transform Expression Semantics
  • Added the pipeline.transform.expression.semantics option.
  • Added LEGACY as the default value to preserve backward compatibility.
  • Added FLINK_SQL as an opt-in mode for Flink SQL-compatible expression evaluation.
  • Propagated the configured semantics through the composer and Transform runtime.
  1. Flink SQL-Compatible Predicate Semantics
  • Added SQL three-valued logic for =, <>, <, <=, >, and >=.
  • Added SQL-compatible BETWEEN / NOT BETWEEN and IN / NOT IN behavior.
  • Represents SQL UNKNOWN as a nullable Boolean.
  • Changed two-argument LIKE / NOT LIKE to use SQL % and _ wildcards with whole-string matching.
  • Keeps rows in Transform filters only when the result is TRUE, while projections preserve UNKNOWN as NULL.
  1. Test Coverage and Documentation
  • Added runtime unit tests for both legacy and Flink SQL semantics.
  • Added YAML Transform specs covering comparison, BETWEEN, IN, LIKE, and row-level filtering with UNKNOWN.
  • Added YAML configuration parsing coverage for the new option and its default value.
  • Documented the configuration and behavioral differences in both English and Chinese.

JIRA Reference

https://issues.apache.org/jira/browse/FLINK-40203

@github-actions github-actions Bot added docs Improvements or additions to documentation composer common runtime cli labels Jul 23, 2026
@haruki-830
haruki-830 marked this pull request as ready for review July 23, 2026 06:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli common composer docs Improvements or additions to documentation runtime

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant