Skip to content
This repository has been archived by the owner on Dec 14, 2022. It is now read-only.

[BUG] DeserializationSchema::isEndOfStream does not terminate FlinkPulsarSource #109

Closed
enriched opened this issue Jul 23, 2020 · 1 comment · Fixed by #95
Closed

[BUG] DeserializationSchema::isEndOfStream does not terminate FlinkPulsarSource #109

enriched opened this issue Jul 23, 2020 · 1 comment · Fixed by #95
Assignees
@jiazhai @jianyun8023
Labels
triage/week-30 type/bug

Comments

@enriched
Copy link

Describe the bug
When using a Flink DeserializationSchema that has overridden the isEndOfStream method, the source does not terminate when isEndOfStream evaluates to true.

To Reproduce
Steps to reproduce the behavior:

  1. Create a DeserializationSchema that returns true for isEndOfStream
  2. Use FlinkPulsarSource with the schema
  3. Stream continues indefinitely

Expected behavior
The FlinkPulsarSource closes with a maximum timestamp to flush downstream nodes.

Additional context
I am currently attempting to use this in a very simple flink job to export event-time-bounded selections data from pulsar into an S3 bucket. The job will not finalize the file till the stream terminates. If I manually cancel the stream the data shows up in s3.
@hipstern
Copy link

I'm getting the exact same problem.

@jianyun8023 jianyun8023 self-assigned this Aug 12, 2020
@jianyun8023 jianyun8023 linked a pull request Aug 12, 2020 that will close this issue
[FEATURE] adding an interface for pulsar deserialization schema #95
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@jiazhai jiazhai

@jianyun8023 jianyun8023

Labels
triage/week-30 type/bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[FEATURE] adding an interface for pulsar deserialization schema wuzhanpeng/pulsar-flink
5 participants
@sijie @hipstern @jiazhai @enriched @jianyun8023