Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adapter模式丢数据的风险 #4864

Closed
agapple opened this issue Sep 4, 2023 · 1 comment
Closed

adapter模式丢数据的风险 #4864

agapple opened this issue Sep 4, 2023 · 1 comment
Milestone

Comments

@agapple
Copy link
Member

agapple commented Sep 4, 2023

adapter大致的调度结构:

canalMsgConsumer.connect();
while (running) {
     for (int i = 0; i < retry; i++) {
           canalMsgConsumer.getMessage() ;
           canalMsgConsumer.ack / rollback(); 
     }
}
canalMsgConsumer.disconnect();

这里会在adapter running退出时执行disconnect逻辑,目前部分的connector组件的disconnect逻辑有unsubscribe的行为
比如,以CanalTCPConsumer为例:

public void disconnect() {
        canalConnector.unsubscribe();
        canalConnector.disconnect();
    }
}
tcp模式下unsubscribe发送给canal-server,会导致清理cursor位点,如果此时canal-server出现重启,就会从最新的binlog开始消费,从而binlog出现数据丢失的风险
@agapple
Copy link
Member Author

agapple commented Sep 4, 2023

kafka/rocketmq,在disconnect后还能继续保持上一个位点继续消费,暂时不影响

@agapple agapple closed this as completed Sep 4, 2023
@agapple agapple added this to the v1.1.7 milestone Sep 4, 2023
zoemak pushed a commit to zoemak/canal that referenced this issue Jan 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant