[FLINK-26712][table-planner] Metadata keys should not conflict with physical columns #19236
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hackergin
reviewed
Mar 27, 2022
| @@ -99,6 +99,25 @@ class TableScanTest extends TableTestBase { | |||
| util.verifyExecPlan("SELECT * FROM MetadataTable") | |||
| } | |||
|
|
|||
| @Test | |||
| def testDDLWithMetadataThatConflictsWithPhysicalColumn(): Unit = { | |||
There was a problem hiding this comment.
Should the same test be done for the sink table as well?
matriv
reviewed
Mar 31, 2022
| @@ -71,6 +71,9 @@ | |||
| @Internal | |||
| public final class DynamicSourceUtils { | |||
|
|
|||
| // Ensures that physical and metadata columns don't collide. | |||
| public static final String METADATA_COLUMN_PREFIX = "$metadata$"; | |||
There was a problem hiding this comment.
minor, but maybe we can define this constant in a common place for both source and sink?
There was a problem hiding this comment.
the source and sink stack are completely separate, that's why we also have separate util classes
| IntStream.range(0, metadata.size()) | ||
| .mapToObj( | ||
| i -> { | ||
| // Access metadata columns from behind because the |
There was a problem hiding this comment.
from behind -> backwards, or in reverse ?
There was a problem hiding this comment.
I looked it up and chose from the back
| @@ -99,6 +99,25 @@ class TableScanTest extends TableTestBase { | |||
| util.verifyExecPlan("SELECT * FROM MetadataTable") | |||
| } | |||
|
|
|||
| @Test | |||
| def testDDLWithMetadataThatConflictsWithPhysicalColumn(): Unit = { | |||
twalthr
added a commit
to twalthr/flink
that referenced
this pull request
Apr 4, 2022
…hysical columns This reduces the likelihood for name collisions between metadata columns and physical columns. It might break some connector implementations that used SupportsReadable/WritingMetadata and name-based column arithmetics. We don't recommend using name-based column arithmetics but index-based ones. This closes apache#19236.
twalthr
added a commit
that referenced
this pull request
Apr 4, 2022
…hysical columns This reduces the likelihood for name collisions between metadata columns and physical columns. It might break some connector implementations that used SupportsReadable/WritingMetadata and name-based column arithmetics. We don't recommend using name-based column arithmetics but index-based ones. This closes #19236.
JasonLeeCoding
pushed a commit
to JasonLeeCoding/flink
that referenced
this pull request
May 27, 2022
…hysical columns This reduces the likelihood for name collisions between metadata columns and physical columns. It might break some connector implementations that used SupportsReadable/WritingMetadata and name-based column arithmetics. We don't recommend using name-based column arithmetics but index-based ones. This closes apache#19236.
czy006
pushed a commit
to czy006/flink
that referenced
this pull request
Jun 30, 2022
…hysical columns This reduces the likelihood for name collisions between metadata columns and physical columns. It might break some connector implementations that used SupportsReadable/WritingMetadata and name-based column arithmetics. We don't recommend using name-based column arithmetics but index-based ones. This closes apache#19236.
czy006
pushed a commit
to czy006/flink
that referenced
this pull request
Jun 30, 2022
…hysical columns This reduces the likelihood for name collisions between metadata columns and physical columns. It might break some connector implementations that used SupportsReadable/WritingMetadata and name-based column arithmetics. We don't recommend using name-based column arithmetics but index-based ones. This closes apache#19236.
zstraw
pushed a commit
to zstraw/flink
that referenced
this pull request
Jul 4, 2022
…hysical columns This reduces the likelihood for name collisions between metadata columns and physical columns. It might break some connector implementations that used SupportsReadable/WritingMetadata and name-based column arithmetics. We don't recommend using name-based column arithmetics but index-based ones. This closes apache#19236.
czy006
pushed a commit
to czy006/flink
that referenced
this pull request
Jul 7, 2022
…hysical columns This reduces the likelihood for name collisions between metadata columns and physical columns. It might break some connector implementations that used SupportsReadable/WritingMetadata and name-based column arithmetics. We don't recommend using name-based column arithmetics but index-based ones. This closes apache#19236.
czy006
pushed a commit
to czy006/flink
that referenced
this pull request
Jul 11, 2022
…hysical columns This reduces the likelihood for name collisions between metadata columns and physical columns. It might break some connector implementations that used SupportsReadable/WritingMetadata and name-based column arithmetics. We don't recommend using name-based column arithmetics but index-based ones. This closes apache#19236.
czy006
pushed a commit
to czy006/flink
that referenced
this pull request
Jul 15, 2022
…hysical columns This reduces the likelihood for name collisions between metadata columns and physical columns. It might break some connector implementations that used SupportsReadable/WritingMetadata and name-based column arithmetics. We don't recommend using name-based column arithmetics but index-based ones. This closes apache#19236.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
This reduces the likelihood for name collisions between metadata columns and physical columns. It might break some connector implementations that used
SupportsReadable/WritingMetadataand name-based column arithmetics. Since this might only be done by very advanced connector implementations (such as our FileSystem connector) and we don't recommend using name-based column arithmetics but index-based ones. This change should not affeect many users and should be done asap as it changes the JSON plan.Brief change log
$metadata$Verifying this change
This change added tests and can be verified as follows:
TableScanTestDoes this pull request potentially affect one of the following parts:
@Public(Evolving): yesDocumentation