Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ARCTIC-1171][FLINK] When reading partial fields from Logstore, the number of fields does not match #1172

Merged
merged 7 commits into from
Mar 1, 2023

Conversation

lklhdu
Copy link
Contributor

@lklhdu lklhdu commented Feb 28, 2023

Why are the changes needed?

fix #1171

Brief change log

  • LogDynamicSource support ProjectionPushDown feature
  • The flink 1.12, 1.14 and 1.15 versions are affected.

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduces a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@github-actions github-actions bot added the module:mixed-flink Flink moduel for Mixed Format label Feb 28, 2023
@lklhdu lklhdu changed the title [ARCTIC-1171][FLINK] When reading some fields from Logstore, the number of fields does not match [ARCTIC-1171][FLINK] When reading partial fields from Logstore, the number of fields does not match Feb 28, 2023
Copy link
Contributor

@YesOrNo828 YesOrNo828 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lklhdu Generally speaking, this patch is ok, there is a small place that needs to modify.

@@ -150,6 +162,14 @@ protected LogPulsarSource createPulsarSource() {
projectedSchema = new Schema(Arrays.stream(valueProjection).mapToObj(columns::get).collect(Collectors.toList()));
}

if (projectedFields != null) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about abstracting the common logic to a private method?

Copy link
Contributor

@YesOrNo828 YesOrNo828 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@YesOrNo828 YesOrNo828 merged commit bf9880e into apache:master Mar 1, 2023
YesOrNo828 pushed a commit that referenced this pull request Mar 1, 2023
…umber of fields does not match (#1172)

LogDynamicSource support projected push down.

(cherry picked from commit bf9880e)
@lklhdu lklhdu deleted the bugfix branch March 7, 2023 06:41
zhoujinsong pushed a commit that referenced this pull request May 31, 2023
…umber of fields does not match (#1172)

LogDynamicSource support projected push down.
ShawHee pushed a commit to ShawHee/arctic that referenced this pull request Dec 29, 2023
…umber of fields does not match (apache#1172)

LogDynamicSource support projected push down.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
module:mixed-flink Flink moduel for Mixed Format
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug]: When reading partial fields from Logstore, the number of fields does not match
2 participants