Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FLINK-33169] Consider descriptor information during system column expansion #23477

Merged
merged 1 commit into from
Sep 30, 2023

Conversation

twalthr
Copy link
Contributor

@twalthr twalthr commented Sep 29, 2023

What is the purpose of the change

"TABLE t" is translated into an implicit "SELECT * FROM t". This would ignore columns
that are not expanded by default. However, the descriptor explicitly states the need
for this column.

Brief change log

Explicit table expressions (for window TVFs at most one)
are captured before rewriting and replaced with a "marker" SqlSelect that contains the
descriptor information. The "marker" SqlSelect is considered during column expansion.

Verifying this change

This change added tests and can be verified as follows: ColumnExpansionTest

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable

@flinkbot
Copy link
Collaborator

flinkbot commented Sep 29, 2023

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

}
final SqlFunction function = (SqlFunction) call.getOperator();

if (function.getFunctionType() != SqlFunctionCategory.USER_DEFINED_TABLE_FUNCTION) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are you looking only at UDFs (USER_DEFINED_TABLE_FUNCTION) here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For performance, I want to exit this method early. And currently all PTFs have this catagory.

@twalthr twalthr merged commit b25b57c into apache:master Sep 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants