Feature idea: KVS pub sub

[ghost]

The KVS is a generic bucket store.
I was thinking of extending it to publish changes over GRPC.

Use Case ?
When you build Clients using this system, its very useful to know when anything has change and he nature of the change. Then as a subscriber i can update my many microservices or even gui client using this. It keeps them all in sync basically.

The event would be like:

• namespace: the bucket namespace
• eventtype: Create/Read/Update/Delete
• data: protobuf or json.

Also should have the ability to turn off eventtype Read because no one normally needs that, but it can be useful for dynamically knowing who is reading where.

Implementation:
GRPC Middleware might be the perfect fit !
https://github.com/grpc-ecosystem/go-grpc-middleware
Also great for adding other things like security etc.

Because its GRPC, it will be easy to then recieve it and put it onto NATS message broker later also as a 2nd bit of work.

Index.
I thought also about the index but thinks its not worth the effort. What you can do is make each index query output to a cache into the KVS store. then you can use PUB SUB from that. Almost like a Materialized View with pub sub on top of it. And also gives you caching for free to a degree.

[mosuka]

Hi gedw99,

Thank you for your feature ideas.
Pub/Sub is interesting. Unfortunately I'm familiar with it so I'll learn and think about it.
gRPC middleware is also interesting. I'll implement this little by little. :)

[ghost]

hey @mosuka thanks for considering it.

I am working on getting the Change to a Bucket to be put onto NATS. This will mena its easy to then update users when any KV bucket changes and the nature of the change.
NATS has adapters for different client transports and also security.

I can send you links if you want.

[mosuka]

@gedw99
I thought KVS would be better with an independent repository.
I decided to spin off into independent repository.
This issue will be transferred to that repository.
See following URL:
mosuka/cete#1

[mosuka]

@gedw99 I do not know if I can implement it, but I'm interested. Can you post the link to the following issue? :)
mosuka/cete#1

[ghost]

closing since its over in cete now

[ghost]

I just had a brain wave about another way to do this.

The raft Wal log !

We can listen to it and see what mutations are occuring

Do you have any links to an API to listen to the RAFT log ?

This btw is also how this is done with databases. The whole area is call CDC ( change data capture ). It's a reliable way because you can be sure it's committed transactionally

[mosuka]

It is an interesting idea.
I'll consider to implement CDC.
Thanks!

[ghost]

I noticed you added a watch ability to the Federation branch.
https://github.com/mosuka/blast/blob/federation/protobuf/management/management.proto#L36
This is intended to get notified of changes i presume ?

I am in the middle of some other work, but will try to get time to try some things out soon.

[mosuka]

Hi,
It is a monitor function that watching event of the cluster like adding / removing nodes.
Similarly, I will add a monitor to the indexer that watching the event of the index like adding / deleting documents.

[ghost]

ok makes sense for the Nodes. Very cool.

Regarding watching for Index changes, your approach is good.
The reason why i suggested watching Raft Log is because its more generic and we can use it for Blast and Cete and maybe other stuff.
A the Raft Log level, you have better ordering and consensus too i expect.

Then we could surface it in the GRPC API. This is a good thing to examine the aspects of it:
https://www.cockroachlabs.com/docs/stable/change-data-capture.html

I wish i had time to dive into the code, but i don't so just writing it down here.

[ghost]

hey @mosuka

Did you see this !!
Badger has Subscriptions now !
dgraph-io/badger#684