Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug][Flink]: Watermark may loss if split read too fast. #486

Closed
1 task done
zstraw opened this issue Oct 19, 2022 · 0 comments · Fixed by #523 or #538
Closed
1 task done

[Bug][Flink]: Watermark may loss if split read too fast. #486

zstraw opened this issue Oct 19, 2022 · 0 comments · Fixed by #523 or #538
Assignees
Labels
module:mixed-flink Flink moduel for Mixed Format type:bug Something isn't working
Milestone

Comments

@zstraw
Copy link
Contributor

zstraw commented Oct 19, 2022

What happened?

If there is an eventTime field defined in arctic table by 'create table t (wm as cast(op_time as timestamp(3)), watermark for wm as wm) like XXX' , watermark may not emitted.

Affects Versions

<=0.3.2

What engines are you seeing the problem on?

Flink

How to reproduce

  1. Define a watermark by sql 'create table t (wm as cast(op_time as timestamp(3)), watermark for wm as wm) like XXX'.
  2. Then group by tumble window, or other sql need watermark.
  3. Insert into source table with one record.

You may find that there is no watermark in Flink web ui.

Relevant log output

No response

Anything else

The cause may be the case like:
The watermark in {@link WatermarkOutputMultiplexer.OutputState} which is associated with a split has been updated, but watermark may not been emitted for that when {@link WatermarkOutputMultiplexer#onPeriodicEmit} called, the outputState has been removed by {@link WatermarkOutputMultiplexer#unregisterOutput(String)} after split finished.

Code of Conduct

  • I agree to follow this project's Code of Conduct
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
module:mixed-flink Flink moduel for Mixed Format type:bug Something isn't working
Projects
None yet
1 participant