MySQL Binlog subscriber

[roblaszczak]

MySQL binlog subscriber may be useful for implementing tool like Debezium.

To check before implementation:

• https://github.com/siddontang/go-mysql-elasticsearch
• https://github.com/siddontang/go-mysql (used in go-mysql-elasticsearch)
• https://github.com/uber/storagetapper
• https://github.com/samsarahq/thunder/blob/v0.5.0/livesql/binlog.go

This implementation should be added in https://github.com/ThreeDotsLabs/watermill-sql/ repository. It need to be compatible with already existing SQL Subscriber.

That means, that you should use SQL publisher for publishing messages, and binlog subscriber to read them.
We would, of course, keep already existing SQL based subscriber.

[njern]

Do you guys have thoughts around tapping into the PostgreSQL equivalent (a WAL subscriber) ?

[m110]

I guess logical decoding could be used, as in debezium: https://debezium.io/docs/connectors/postgresql/
Although I'm not sure if there is a library for Go at the moment.

[joeblew99]

@roblaszczak
i would not use any of the Deebzium stuff. I dont like its approach.

for MySQL use this: https://github.com/uber/storagetapper

for PostreSQL use thunder: https://github.com/samsarahq/thunder/blob/v0.5.0/livesql/binlog.go

let me know if your curious

[roblaszczak]

@joeblew99 for now simple sql-queries based subscriber is available: https://github.com/ThreeDotsLabs/watermill-sql

We have in plans add also Binlog based subscriber, but we need more time and be sure that this simple implementation is stable :)

[joeblew99]

I had a look at https://github.com/ThreeDotsLabs/watermill-sql Its a bit obtuse as to what its doing exactly. It seems to be designed for polling the DB to see what changed ? - wow its brute force - it solves the issue of having the parse the query and match it to the topic based on the predicates which is a tricks problem.

…

On Tue, Aug 13, 2019 at 8:33 PM Robert Laszczak ***@***.***> wrote: @joeblew99 <https://github.com/joeblew99> for now simple sql-queries based subscriber is available: https://github.com/ThreeDotsLabs/watermill-sql We have in plans add also Binlog based subscriber, but we need more time and be sure that this simple implementation is stable :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5?email_source=notifications&email_token=AC3RU43HVFBBST2BOUXR7C3QEL5BHA5CNFSM4GE72FB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4GSJXI#issuecomment-520955101>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AC3RU472HFQGCDEY4DASG43QEL5BHANCNFSM4GE72FBQ> .

[roblaszczak]

@joeblew99 so creating docs is in progress so it may be not clear, but in short:

• you are able to produce and consume messages from MySQL (but you can extend it for other SQL's if you need) tables
• it's using just SQL queries, it's slow but fast enough for a lot of projects (for projects where it's not - we will introduce binlog implementation
• and it's of course fully compatible with Watermill's Pub/Sub, so you can use router, middlewares etc.

[joeblew99]

@roblaszczak Thanks and this all makes sense.

[0michalsokolowski0]

I would like to work on it

[Jonathan34]

@0michalsokolowski0 wondering if you made any promising progress on that? i am probably fine with mysql select queries but just wondering