Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat][jdbc-#1300]add parameter pollingFromMax #1301

Merged
merged 2 commits into from
Oct 18, 2022

Conversation

Paddy0523
Copy link
Contributor

… pollingMode starts with the maximum value of the incrementColumn

Purpose of this pull request

Set the parameter 'pollingFromMax=true' so that the pollingMode starts with the maximum value of the incrementColumn

Which issue you fix

Fixes # (issue).
#1300

Checklist:

  • I have executed the 'mvn spotless:apply' command to format my code.
  • I have a meaningful commit message (including the issue id, the template of commit message is '[label-type-#issue-id][fixed-module] a meaningful commit message.')
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have made corresponding changes to the documentation.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have checked my code and corrected any misspellings.
  • My commit is only one. (If there are multiple commits, you can use 'git squash' to compress multiple commits into one.)

…hat the pollingMode starts with the maximum value of the incrementColumn
@Paddy0523 Paddy0523 added the feature-request this is a feature requests on the product label Oct 12, 2022
@Paddy0523 Paddy0523 added this to the 1300 milestone Oct 12, 2022
@Paddy0523 Paddy0523 removed this from the 1300 milestone Oct 12, 2022
@Paddy0523
Copy link
Contributor Author

@baisui1981 @FlechazoW

String maxValue;
if (inputSplit.getSplitNumber() == 0) {
maxValue = getMaxValueFromDb();
maxValue = incrementKeyUtil.transToLocationValue(getMaxValueFromDb()).toString();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here, I have a concern about whether the returned maxval value should be+1 when pollingFromMax=true. Otherwise, the last item in the history will be consumed when consumption starts (it should not be consumed)

这里就是,我有一个担心,就是 当pollingFromMax=true 时候,这个返回的 maxval值是否要+1 ,不然,开始消费的时候 会把历史记录的最后一条也消费的(应该是不消费的 )
我今天自己的实现 是 用了一个 这样的方法 https://github.com/qlangtech/chunjun/blob/61a1f09bd31bcb3c7528a55178a835db6d4e81ff/chunjun-connectors/chunjun-connector-jdbc-base/src/main/java/com/dtstack/chunjun/connector/jdbc/source/JdbcInputFormat.java#L196,

我之前测试的时候 是用 “where #column >= #val” 的,改过了?其实会将历史最后一条也消费的(其实并不需要)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

useMaxFunc and pollingFromMax cannot be used together

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

useMaxFunc and pollingFromMax cannot be used together

Could you please tell me the reason. In my opinion, the param 'pollingFromMax' and 'useMaxFunc' is not conflict with each other.

Copy link
Contributor Author

@Paddy0523 Paddy0523 Oct 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please tell me the reason. In my opinion, the param 'pollingFromMax' and 'useMaxFunc' is not conflict with each other.

Can you list how you expect the program to behave when 'pollingFromMax' and 'useMaxFunc' are set to various values (there should be four combinations here)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can imagine that if polling keeps consuming the last piece of data (>=), does it mean that the last piece of data may be consumed multiple times
Why don't we just use '>' instead of '+1' if we don't want to consume the last piece of data over and over again (' +1 'doesn't work for double data in my opinion)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok ,I see ,you are right, thanks

@FlechazoW FlechazoW merged commit 41e6489 into DTStack:master Oct 18, 2022
lyzeo pushed a commit to lyzeo/chunjun that referenced this pull request Oct 19, 2022
…hat the pollingMode starts with the maximum value of the incrementColumn (DTStack#1301)

Co-authored-by: FlechazoW <35768015+FlechazoW@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request this is a feature requests on the product
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature][connector-jdbc-base] avoid consume historical data from db to consume latest db updates events
3 participants