Skip to content

[mongodb] Add disableCursorTimeout option, addresses #2302#2332

Merged
Jiabao-Sun merged 1 commit intoapache:masterfrom
yuxiqian:mongo/cursor-timeout-config
Jul 24, 2023
Merged

[mongodb] Add disableCursorTimeout option, addresses #2302#2332
Jiabao-Sun merged 1 commit intoapache:masterfrom
yuxiqian:mongo/cursor-timeout-config

Conversation

@yuxiqian
Copy link
Copy Markdown
Member

This PR fixes issue #2302.

Currently, MongoDB CDC uses noCursorTimeout option when performing incremental snapshot. However, some MongoDB server doesn't support creating cursor with unlimited timeout.

To address this issue, an option disableCursorTimeout was added in mongodb.source.MongoDBSource (but not its legacy counterpart mongodb.MongoDBSource since it isn't affected).

@yuxiqian
Copy link
Copy Markdown
Member Author

Haven't added it into Table API since I noticed that DataSource API seems to have more "hidden" (undocumented) options than Table API. Plus, this option only takes into effect if scan.incremental.snapshot is set to true.

@yuxiqian yuxiqian force-pushed the mongo/cursor-timeout-config branch 3 times, most recently from b3de4dc to 1f59e64 Compare July 21, 2023 03:33
@yuxiqian
Copy link
Copy Markdown
Member Author

Could @Jiabao-Sun help reviewing it?

@Jiabao-Sun Jiabao-Sun self-requested a review July 24, 2023 02:33
@Jiabao-Sun
Copy link
Copy Markdown
Contributor

Thanks @yuxiqian for this enhancement.
Shall we add it as a table option as well?

@yuxiqian yuxiqian force-pushed the mongo/cursor-timeout-config branch from 1f59e64 to 0f90eae Compare July 24, 2023 04:11
@yuxiqian yuxiqian marked this pull request as ready for review July 24, 2023 04:11
@yuxiqian
Copy link
Copy Markdown
Member Author

Added. Picked name scan.cursor.no-timeout from Flink MongoDB Connector (non-CDC version).

Copy link
Copy Markdown
Contributor

@Jiabao-Sun Jiabao-Sun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @yuxiqian for the updates. LGTM.

@Jiabao-Sun Jiabao-Sun merged commit a8dd4d3 into apache:master Jul 24, 2023
GOODBOY008 pushed a commit to GOODBOY008/flink-cdc that referenced this pull request Oct 30, 2023
GOODBOY008 pushed a commit to GOODBOY008/flink-cdc that referenced this pull request Oct 30, 2023
ChaomingZhangCN pushed a commit to ChaomingZhangCN/flink-cdc that referenced this pull request Jan 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants