Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[INLONG-10560][Sort] Support bounded pulsar source #10569

Merged

Conversation

aloyszhang
Copy link
Contributor

@aloyszhang aloyszhang commented Jul 5, 2024

Fixes #10560

Motivation

The pulsar source only supports un-bounded data sources, this pull request aims to add the support for bounded pulsar source.

Modifications

  1. define the boundaries for the source
  2. make pulsar source support bounded data(this pull request only support time boundaries)

This is worked for offline sync jobs with Flink batch runtime mode.
Every time the Flink batch task begins, set the start and stop timestamp(the publish time of pulsar msg) of the Pulsar source.
The time to finish the batch Flink job is determined by the stop boundary(StopCursor for the Pulsar source).

Note: since the boundaries, especially the stop boundary, are defined by the pulsar message's publish time, the batch Flink job may not be finished if no messages are published into the Pulsar topic.

Verifying this change

(Please pick either of the following options)

  • This change is a trivial rework/code cleanup without any test coverage.

Documentation

  • Does this pull request introduce a new feature? ( no)

Copy link
Contributor

@baomingyu baomingyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@aloyszhang aloyszhang merged commit bf38d2a into apache:dev-offline-sync Jul 5, 2024
10 checks passed
@aloyszhang aloyszhang self-assigned this Jul 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants