-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[feat][jdbc-#1300]add parameter pollingFromMax #1301
Conversation
…hat the pollingMode starts with the maximum value of the incrementColumn
String maxValue; | ||
if (inputSplit.getSplitNumber() == 0) { | ||
maxValue = getMaxValueFromDb(); | ||
maxValue = incrementKeyUtil.transToLocationValue(getMaxValueFromDb()).toString(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here, I have a concern about whether the returned maxval value should be+1 when pollingFromMax=true. Otherwise, the last item in the history will be consumed when consumption starts (it should not be consumed)
这里就是,我有一个担心,就是 当pollingFromMax=true 时候,这个返回的 maxval值是否要+1 ,不然,开始消费的时候 会把历史记录的最后一条也消费的(应该是不消费的 )
我今天自己的实现 是 用了一个 这样的方法 https://github.com/qlangtech/chunjun/blob/61a1f09bd31bcb3c7528a55178a835db6d4e81ff/chunjun-connectors/chunjun-connector-jdbc-base/src/main/java/com/dtstack/chunjun/connector/jdbc/source/JdbcInputFormat.java#L196,
我之前测试的时候 是用 “where #column >= #val” 的,改过了?其实会将历史最后一条也消费的(其实并不需要)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
useMaxFunc and pollingFromMax cannot be used together
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
useMaxFunc and pollingFromMax cannot be used together
Could you please tell me the reason. In my opinion, the param 'pollingFromMax' and 'useMaxFunc' is not conflict with each other.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you please tell me the reason. In my opinion, the param 'pollingFromMax' and 'useMaxFunc' is not conflict with each other.
Can you list how you expect the program to behave when 'pollingFromMax' and 'useMaxFunc' are set to various values (there should be four combinations here)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can imagine that if polling keeps consuming the last piece of data (>=), does it mean that the last piece of data may be consumed multiple times
Why don't we just use '>' instead of '+1' if we don't want to consume the last piece of data over and over again (' +1 'doesn't work for double data in my opinion)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok ,I see ,you are right, thanks
…hat the pollingMode starts with the maximum value of the incrementColumn (DTStack#1301) Co-authored-by: FlechazoW <35768015+FlechazoW@users.noreply.github.com>
… pollingMode starts with the maximum value of the incrementColumn
Purpose of this pull request
Set the parameter 'pollingFromMax=true' so that the pollingMode starts with the maximum value of the incrementColumn
Which issue you fix
Fixes # (issue).
#1300
Checklist: