Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MySQL Binlog subscriber #5

Open
roblaszczak opened this issue Nov 18, 2018 · 9 comments
Open

MySQL Binlog subscriber #5

roblaszczak opened this issue Nov 18, 2018 · 9 comments

Comments

@roblaszczak
Copy link
Member

@roblaszczak roblaszczak commented Nov 18, 2018

MySQL binlog subscriber may be useful for implementing tool like Debezium.

To check before implementation:

This implementation should be added in https://github.com/ThreeDotsLabs/watermill-sql/ repository. It need to be compatible with already existing SQL Subscriber.

That means, that you should use SQL publisher for publishing messages, and binlog subscriber to read them.
We would, of course, keep already existing SQL based subscriber.

@roblaszczak roblaszczak changed the title MySQL Binlog MySQL Binlog subscriber Nov 18, 2018
@njern
Copy link

@njern njern commented Dec 3, 2018

Do you guys have thoughts around tapping into the PostgreSQL equivalent (a WAL subscriber) ?

@m110
Copy link
Member

@m110 m110 commented Dec 4, 2018

I guess logical decoding could be used, as in debezium: https://debezium.io/docs/connectors/postgresql/
Although I'm not sure if there is a library for Go at the moment.

@joeblew99
Copy link

@joeblew99 joeblew99 commented Aug 13, 2019

@roblaszczak
i would not use any of the Deebzium stuff. I dont like its approach.

for MySQL use this: https://github.com/uber/storagetapper

for PostreSQL use thunder: https://github.com/samsarahq/thunder/blob/v0.5.0/livesql/binlog.go

let me know if your curious

@roblaszczak
Copy link
Member Author

@roblaszczak roblaszczak commented Aug 13, 2019

@joeblew99 for now simple sql-queries based subscriber is available: https://github.com/ThreeDotsLabs/watermill-sql

We have in plans add also Binlog based subscriber, but we need more time and be sure that this simple implementation is stable :)

@joeblew99
Copy link

@joeblew99 joeblew99 commented Aug 13, 2019

@roblaszczak
Copy link
Member Author

@roblaszczak roblaszczak commented Aug 13, 2019

@joeblew99 so creating docs is in progress so it may be not clear, but in short:

  • you are able to produce and consume messages from MySQL (but you can extend it for other SQL's if you need) tables
  • it's using just SQL queries, it's slow but fast enough for a lot of projects (for projects where it's not - we will introduce binlog implementation
  • and it's of course fully compatible with Watermill's Pub/Sub, so you can use router, middlewares etc.
@joeblew99
Copy link

@joeblew99 joeblew99 commented Aug 14, 2019

@roblaszczak Thanks and this all makes sense.

@0michalsokolowski0
Copy link
Contributor

@0michalsokolowski0 0michalsokolowski0 commented Oct 10, 2019

I would like to work on it

@Jonathan34
Copy link

@Jonathan34 Jonathan34 commented Nov 8, 2019

@0michalsokolowski0 wondering if you made any promising progress on that? i am probably fine with mysql select queries but just wondering

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants