-
Notifications
You must be signed in to change notification settings - Fork 13.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FLINK-20947][planner] Fix idle source doesn't work when pushing watermark into the source #14679
Conversation
…rmark into the source
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community Automated ChecksLast check on commit d21a2ca (Fri May 28 07:12:32 UTC 2021) Warnings:
Mention the bot in a comment to re-run the automated checks. Review Progress
Please see the Pull Request Review Guide for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commandsThe @flinkbot bot supports the following commands:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the work.
I think we need an IT case for Kafka to verify the pushed idle watermark strategy can work well. You can add a test in KafkaTableITCase
with:
- prepare 2 partitions, each with several messages. The last event time in the first partition is 2021-01-13 20:00:00, the last event time in the second partition is 2021-01-13 19:30:00.
- apply window query on the source with tumbling 1 hour
- the query should have output with window [19:00, 20:00) which means the watermark has reached 20:00 (before the fix, the query shouldn't have output)
If we have an IT case for Kafka, I think we don't need to adapt the TestValues source. The changes looks very complex and hard to maintain in the future.
Thanks for your tips. I think I have already resolved your concerns. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great work! The pull request looks good to me now.
Could you also open a pull request for release-1.12 branch?
…rmark into the source This closes apache#14679
…rmark into the source This closes apache#14679
What is the purpose of the change
Fix idle source doesn't work when pushing watermark strategy into table source.
Brief change log
Verifying this change
This change added tests and can be verified as follows:
Does this pull request potentially affect one of the following parts:
@Public(Evolving)
: (yes / no)Documentation