Replies: 1 comment
-
This is generally a great post, and the information about DD and MZ is very sharp. There's one issue that I wanted to call out though! You say, in the section "Overheads of Remote Accessing State", that
However, there is zero remote access to arrangements in differential dataflow. The code literally does not exist. I think the misunderstanding can be traced back to
which supposes that arrangements are made on nodes. In fact, arrangements are sharded across nodes, by key. So the arrangements of tables A and B will have a fraction of their keyspace on Node 1, and another fraction on Node 2. As long as the sharding by key is consistent, each of Nodes 1 and 2 will have the right subsets of A and B locally, and can implement a join with no further data exchange or remote access. |
Beta Was this translation helpful? Give feedback.
-
https://skyzh.dev/blog/2023-12-28-store-of-streaming-states/
Beta Was this translation helpful? Give feedback.
All reactions