Increase concurrency handling while refetching state

[0x777]

No description provided.

[nicosuave]

Does this currently blindly poll every second + cache results in memory? https://github.com/hasura/graphql-engine/blob/master/server/src-lib/Hasura/GraphQL/Resolve/LiveQuery.hs#L112

[nicosuave]

Just spitballing, but might be worth tracking query dependencies (tables), as well as building a dependency graph of tables used in views. Then, read the wal log + find up matching queries to execute. Were y'all considering another approach?

[coco98]

@nicosuave We had done something exactly like that earlier but realised that for anything with a high ingestion rate, it crumbles pretty quickly (both WAL and listen/notify) This apart horizontal scaling needs care to manage logical-repl slots etc, . https://github.com/hasura/pgdeltastream https://github.com/hasura/skor

The new update @0x777 is merging is going to be a huge improvement and a different approach. As soon as its merged we'll write up a blog on how it works. It'll allow a few thousand concurrent subscriptions to be multiplexed. We're benchmarking that versus a pure wal/listen-notify approach too. Will link you to it as soon as its out! 🤞 for monday.

[mike-thompson-day8]

If I understand correctly, the current approach (of brute force polling and associated diffing) trades off the cost of substantial CPU usage against the benefits of algorithmic simplicity and relatively low memory use.

Even when subscription queries are multiplexed (by which I assume you mean de-duplicated?), the tradeoff is unchanged (but less CPU and less memory is used in the extent that there's concurrent, duplicate queries).

If you are interested in exploring the alternative trade-off ... less CPU but more RAM, I'd point you towards the research done on "differential dataflow" (originally by Microsoft Research Labds but then carried on at ETH Zurich’s Systems group). There's a RUST implementation here:
https://github.com/TimelyDataflow/differential-dataflow
and a recent talk (assumes knowledge of Datalog-like databases):
https://www.youtube.com/watch?v=ZgqFlowyfTA

As far as I can tell (as a layperson), their work represents the current state of the art.

[ecthiender]

@0x777 @coco98 we can close this right?

[marionschleifer]

I'm closing this issue. It has been fixed by this release: https://github.com/hasura/graphql-engine/releases/tag/v1.0.0-alpha45 🙂