Subscribe to changes to a document/query (live query)

[mitar]

When building web apps, I like to build them with reactivity: I subscribe to live queries on the client and then data behind those queries change, the changes are pushed to the client, which I can then use to update UI accordingly.

With Dolt's architecture I could imagine that something like that should be possible to be implemented, given that it keeps historical data and that it can quickly computes diffs, then when data changes, it could compute a diff and send it to the client.

But I didn't find anything like that yet. So I am making a feature request.

[fulghum]

Hey @mitar, thanks for the feature request. Being able to subscribe to data updates would be a very cool feature. We don't support this currently, but it's an interesting idea. There are some other libraries we know of that serve a similar purpose and might be worth checking out, such as Replicache, which is built on some similar internal technical concepts as Dolt.

[timsehn]

This is a cool feature request.

You can do this now with a polling architecture:

1. Every N seconds, query the HEAD of the branch you are interested in using the dolt_log system table.
2. If unchanged, stop
3. If new commit, store new latest commit.
4. Query dolt_diff system table to see if the tables in the queries you are interested in have changed
5. if no, stop
6. If yes, query dolt_diff_<tablename> to see if the rows you are interested in have changed
7. If no, stop
8. If yes, reexecute your query.

This seems like a lot of steps but the log and diff queries are very fast.

Were you thinking of a different design? Maybe make the queries views and then provide diffs on views?

[mitar]

I have not known Replicache despite being pretty interested in this space, so thank you for sharing it with me. I know there are libraries like that, I have even made my own one for PostgreSQL.

I think polling architecture seems fine and is probably good enough for most cases. I think the architecture you described works well even without polling, if there is an event "this table changed", you can then do similar steps. So I think the first step towards live queries would be to have events when tables change. Then you cold reexecute queries which depend on those tables (this is what my library linked above is doing for PostgreSQL).

But if we are discussing this, I think the best approach is in fact to implement what is known as "incremental materialized view maintenance" algorithm, which is able to know from any cell which changes, which materialized views (but really any query) it has to update and exactly how, without having to rerun the query. See some notes I made around this. I think Dolt has an advantage here because it tracks changes per cell, so it can really know what changed and when. Btu then how does this impact active live queries, this is then a lot of work. So reexecuting queries is generally a reasonable solution.

[timsehn]

We've discussed materialized views which is why I kind of alluded to them :-)

Materialized views are on the roadmap but we're unlikely going to get to them soon.

[mitar]

Yea, materialized views are in my view just special case of live query. :-) Live query where updates are pushed into a view vs. live query where updates are pushed to the client.

[timsehn]

I'm going to close. Thanks for the feedback.