Support logical replication / change data capture (CDC) with Kafka

[fritzblue]

The following workflow is becoming more and more common in streaming architectures:

If Dgraph is to be used as a primary data store in a production system, it must support logical replication / change data capture.

Is this currently possible? If not, is it planned for the future?

[pawanrawal]

It's not possible right now but we can do something like this in the future. Some users have asked for subscriptions. This would be a similar feature I suppose.

[manishrjain]

We could potentially support this at Badger level first, and then expose it via Dgraph.

hypermodeinc/badger#684

[campoy]

After chatting with @gitlw, it seems like this could be implemented relatively easily on top on our current plans for datacenter to datacenter replication, which already uses Kafka.

Adding it to our 1.X plans!

[fritzblue]

For an example of a distributed DB that's doing this in what I would argue is the Right Way™, see CockroachDB https://www.cockroachlabs.com/docs/stable/change-data-capture.html