feat: Introduce MySQL Connector #1829
Conversation
abcpro1
force-pushed
the
mysql
branch
2 times, most recently
from
3da019b
to
aef9633
Compare
|
Thanks for your contribution @abcpro1. Waiting for @chubei to review so we can merge and payout the bounty. @abcpro1 would u be up for a call ? Can drop me a note at matteo@getdozer.io ? |
|
Hi @abcpro1 thank you for your work! The implementation looks amazing and I can't wait to merge it. I hope you can add several more things to this PR. Please let me know what you think.
This method is mostly for documentation purpose, so we can quickly see which types are supported instead of digging into the code.
I see that you've added an end to end test in To add a new connector to the integration test suite is very easy. You implement several traits which does the database setup, and add a test that calls the test suite on your connector. You can take the
I don't see any unit tests yet (or I missed them?). I hope the core logic can be covered by unit tests. |
|
@chubei Thank you for reviewing this pull request. I'm glad you like it. I will add comprehensive tests as you suggested. As for |
Yes. |
|
@abcpro1 it would be great if you can also document in the PR all the config parameters of the connector and maybe add a small sample using MySQL in |
|
@snork-alt Sure. |
Define and implement a MySQL connector for Dozer. This connector operates similarly to the Postgres connector. The replication process starts by replicating rows from MySQL tables using a SELECT query. The next step is using the MySQL CDC protocol (binary log events) for watching live changes to the tables and replicating them in Dozer. All connector requirements are implemented, including listing tables, columns, schemas, and replication. The implementation respects the column and table selection such that only the requested columns are included in the generated schemas and in the replication process; the rest of the columns are ignored.
Define and implement a MySQL connector for Dozer.
It operates similarly to the Postgres connector; especially when it comes to listing tables, columns, and schemas.
For replication, MySQL CDC is used to replay database changes in Dozer, after initially copying the table contents with a simple SELECT query.
The connector implementation is pretty much complete with the only exception of partial updates to JSON fields, which is a feature in MySQL that does not have pre-existing support in Dozer as far as I could tell.
closes #1670
/claim #1670