binh/incremental snapshot keys #4256
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
xinbinhuang
commented
Feb 4, 2023
xinbinhuang
commented
- wip
- Test deploy-file
- clean up
- Pass NOOP operations so that they will be visible downstream
- Pass raw_oplog into Debezium output (DBZ-4 Changed copyright statement in source code headers and adjusted checkstyle rules #3)
- wip (DBZ-6 Enforce line ending style for most file types #4)
- Add debezium-bom (DBZ-5 Added change log file #5)
- Add debug logging for queue size (DBZ-1 Added the MySQL source connector #6)
- Add debug logging in MongoDbStreamingChangeEventSource (DBZ-7 Require Docker 1.9 or higher rather than 1.6 or higher #7)
- Revert "Add debug logging in MongoDbStreamingChangeEventSource (DBZ-7 Require Docker 1.9 or higher rather than 1.6 or higher #7)" (DBZ-1 Refactored the connector modules and improved the embedded engine framework #8)
- Add comment to oplog query (DBZ-10 Added small utility so unit tests can run an embedded Kafka cluster in-process #10)
- Use bytes for raw oplogs (DBZ-11 Build can skip long-running unit and integration tests #11)
- Order imports (add option to run without integration tests #12)
- Use DocumentCodec instead of BsonDocumentCodec (DBZ-1 Completed integration testing and debugging of the MySQL connector #13)
- Enforce RawBsonDocument for oplog capture mode (DBZ-14 Corrected the 'alt-mysql' Maven profile #14)
- Copy bytebuffer into arrary (DBZ-13 Changed Maven build to attach JavaDoc JARs to each module #15)
- Use Bson value directly in JsonSerialization (DBZ-17 Added plugin distribution ZIP for MySQL and other connectors #16)
- Remove extra check on enable oplog flag (DBZ-21 Upgraded to Kafka 0.9.0.1 #17)
- Allow find primary with secondary seed (DBZ-22 Adapted to the Docker Maven Plugin's move to Fabric8 community #18)
- Support skip operation filter on oplog capture mode (DBZ-23 Cleaned up our use of Docker for integration tests #19)
- Allow skipping noop oplogs via skipped.operations config
- Support command op for mongo (#21)
- Add explicit safety check for command operation (Add Google group to README #22)
- Fix getTransactionId
- DBZ-5003 Can't use 'local' database through mongos
- DBZ-5015 Get the json value of a Document when parsing signaling message
- DBZ-5015 Fix formatting
- DBZ-5011 Register MongoDB metrics with task id when more than one task
- DBZ-5260 Filter out unavailable replicaset members
- DBZ-5260 Replace Java 11 code
- Backport upstream changes from debezium-connector-mongodb v1.9.4.Final
- DBZ-5628 Added support for Mongo pre-image in change stream
- DBZ-5628 Add mongo 6.0 version and db version resolver
- DBZ-5628 Run CI with MongoDB 6.0
- DBZ-5628 Add mongo 6.0 pre-image IT test
- DBZ-5628 Fix formatting
- Fix cherry-pick
- Add raw BSON support
- fixup! Add raw BSON support
- fixup! fixup! Add raw BSON support
- Expose heartbeatFrequencyMs for mongo driver (DBZ-30 Changed the MySQL connector to include all columns in the record value #30)
- StripeAudit filtering on oplog mode (DBZ-38 Changed the DDL parser framework to notify listeners as statements are applied #33)
- Check if STRIPE_AUDIT exist first (DBZ-38 Changed the listening framework of the DDL parser #34)
- put toJson in debugMode
- Add metadata.yaml for fork repo (DBZ-42 Use custom mysql images with custom config and startup scripts for integration tests #36)
- Add wallTime in mongo source info (DBZ-48 Cannot parse COMMIT and flush statements #37)
- Expose wallTime in parser (DBZ-29 Changed MySQL connector to be able to hide, truncate, and mask specific columns #38)
- DBZ-5973 Support for more types as key in mongodb incremental snapshot implementation
…_FORCE_DEPLOY Set up Stripe build for Debezium
Co-authored-by: Yang <y.wu4515@gmail.com>
Pass raw_oplog into Debezium output
…ium#7)" (debezium#8) This reverts commit e676035.
Use DocumentCodec instead of BsonDocumentCodec
* Enforce RawBsonDocument for oplog capture mode
As title, because of change in mongo driver Debezium requires we provide primary host name in `mongo.hosts`: https://issues.redhat.com/browse/DBZ-4802 This is also documented in public doc: https://debezium.io/documentation/reference/1.9/connectors/mongodb.html#mongodb-property-mongodb-hosts This PR fixes the problem and allow Debezium to find primary with any seed node in a replset. I tested in QA with config: ``` $ curl -s localhost:28082/connectors/prototype:mongo_shard_test_4_4/config | jq | grep hosts "mongodb.hosts": "shard_test_4_4/10.100.246.141:27017", ``` And verified in the log: ``` [2022-06-02 03:06:20,772] INFO Monitor thread successfully connected to server with description ServerDescription{address=10.100.231.135:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=9, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=2663627, setName='shard_test_4_4', canonicalAddress=10.100.231.135:27017, hosts=[10.100.218.89:27017, 10.100.231.135:27017, 10.100.246.141:27017], passives=[], arbiters=[], primary='10.100.231.135:27017', tagSet=TagSet{[]}, electionId=7fffffff0000000000000043, setVersion=223614, topologyVersion=TopologyVersion{processId=628fc6e5337d7b3e752992a4, counter=21}, lastWriteDate=Thu Jun 02 03:06:20 GMT 2022, lastUpdateTimeNanos=279667569826225} (org.mongodb.driver.cluster) ``` ^ we found the primary `10.100.231.135:27017` automatically (Squashed by Merge Queue - Original PR: https://git.corp.stripe.com/stripe-private-oss-forks/debezium/pull/18)
* Support skip operation filter on oplog capture mode
…skipping-noop-oplog Allow skipping noop oplogs via skipped.operations config
…m/fix-get-transaction-id Fix getTransactionId
MongoDb connector should use "config" database if the database exists. Otherwise, "local" database should be used to get MongoDB version during connector validation. This is similar to the approach implemented in ReplicaSetDiscovery class.
…sync-1.9.4.Final Backport upstream changes from debezium-connector-mongodb v1.9.4.Final
…rry-pick-pre-image cherry-pick: DBZ-5628 Added support for Mongo pre-image in change stream
…-image-base Make Mongo debezium to emit raw BSON
* StripeAudit filtering on oplog mode (debezium#33)
Put the backfill filtering log in debugMode
This is cherry pick debezium@2a366cf from upstream to support wallTime The change has already ran integration test suite in the OSS repo, I also tested it out in QA RS: ``` {"ts": {"$timestamp": {"t": 1674688491, "i": 366}}, "t": 24009, "h": -4248973613377760749, "v": 2, "op": "d", "ns": "locallockdb_9999.merchant_shard_locks", "ui": {"$binary": "eWsvM4XaTze4ne15thV79A==", "$type": 4}, "stripeAudit": "{\"action_id\":\"qa-ingress--0b264e18bbf61bf2d.far-aws-pdx-1.apiori.com/2iqcnsprjF-148443\",\"df_computation\":\"mono-bapi-srv.AccountCreateMethod\",\"host\":\"mono-bapi-srv-p5-green-5f6c4f987b-grkd8.monobapibox.northwest.qa.stripe.io\",\"project\":\"account_abstractions_apis\",\"shard_key\":\"1tfmk21Fwi9\"}", "wall": {"$date": 1674688491324}, "preImageOpTime": {"ts": {"$timestamp": {"t": 1674688491, "i": 365}}, "t": 24009}, "o": {"_id": "mslk_1MUIB5tfmk21Fwi9sJwt7VY5"}} ``` I cannot test CES since 4.4 change event does not have wallTime (Squashed by Merge Queue - Original PR: https://git.corp.stripe.com/stripe-private-oss-forks/debezium/pull/37)
Follow up of https://git.corp.stripe.com/stripe-private-oss-forks/debezium/pull/37 -- make the data available in the parser r? shichao (Squashed by Merge Queue - Original PR: https://git.corp.stripe.com/stripe-private-oss-forks/debezium/pull/38)
|
Welcome as a new contributor to Debezium, @xinbinhuang. Reviewers, please add missing author name(s) and alias name(s) to the COPYRIGHT.txt and Aliases.txt respectively. |
|
Hi @xinbinhuang, thanks for your contribution. Please prefix the commit message(s) with the DBZ-xxx JIRA issue key. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.