Add projections #135
Add projections #135
Conversation
Codecov ReportBase: 90.96% // Head: 89.36% // Decreases project coverage by
Additional details and impacted files@@ Coverage Diff @@
## main #135 +/- ##
==========================================
- Coverage 90.96% 89.36% -1.61%
==========================================
Files 18 19 +1
Lines 3167 3292 +125
==========================================
+ Hits 2881 2942 +61
- Misses 286 350 +64
Flags with carried forward coverage won't be shown. Click here to find out more.
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
the-mikedavis
force-pushed
the
md-projections
branch
from
905a211
to
8ba4f18
Compare
the-mikedavis
force-pushed
the
md-projections
branch
2 times, most recently
from
f14a8be
to
befdaf0
Compare
the-mikedavis
force-pushed
the
md-projections
branch
from
befdaf0
to
08f383b
Compare
the-mikedavis
force-pushed
the
md-projections
branch
3 times, most recently
from
6681e39
to
eeae5b6
Compare
the-mikedavis
force-pushed
the
md-projections
branch
from
eeae5b6
to
09f0aa1
Compare
the-mikedavis
force-pushed
the
md-projections
branch
from
09f0aa1
to
70a38d3
Compare
the-mikedavis
force-pushed
the
md-projections
branch
2 times, most recently
from
17cf48d
to
55c8bd2
Compare
the-mikedavis
force-pushed
the
md-projections
branch
from
55c8bd2
to
b7f9bac
Compare
There was a problem hiding this comment.
I'm so glad about this addition! This is really cool :-)
I submitted several comments. I could split my feedback into two topics:
- Some documentation comments
- The way payloads are passed from the state machine to the projection function
About the payloads, I suggest we do like the rest of the library: call khepri_machine:gather_node_props() to create a node properties map and pass this common format to khepri_projection:trigger(), then decide in that function how to call the projection function.
I think we could pass the entire node properties maps (old and new) to the extended projection function, and reduce payloads to bare data for the simple one (possibly undefined for the old data in case a payload is added). The simple function will never be called when the new payload has no data.
What do you think?
the-mikedavis
force-pushed
the
md-projections
branch
2 times, most recently
from
34d66d0
to
9c99147
Compare
the-mikedavis
force-pushed
the
md-projections
branch
from
9c99147
to
d97caae
Compare
|
Thanks for the feedback @dumbbell! I like the idea of passing For the |
Ra 2.4.0 introduces new reply modes which control which member of a cluster performs the `gen_statem:reply/2` to a command. This can be used to block the caller until the command has been handled by a given member. The child commit will introduce a way to pass the new reply modes in `khepri:command_options()`.
This option wraps the equivalent option in `ra:process_command/3` -
introduced in the dependency upgrade in the parent commit - which
controls which member of the cluster calls `gen_statem:reply/2`. This
can be used to block until a command has been handled by the local
member which can prevent consistency issues with projections. A caller
of `khepri:put/4` might assume that a subsequent `ets:lookup/2` (or
similar) call is up-to-date immediately. By calling `khepri:put/4` with
the `#{reply_from => local}` option, the caller can rely on this
behavior.
the-mikedavis
force-pushed
the
md-projections
branch
3 times, most recently
from
da046b6
to
8cd3925
Compare
Projections are similar in spirit to database views. They are a subset of the store collected into an ETS table which is replicated to all members of the Khepri cluster. Implementation-wise, projections are similar to triggers except that projections are updated synchronously by the machine and are triggered on all changes to a given path pattern. Projections can be used for cases where reads are much more common than writes or in cases where read throughput and/or latency are important. This change initially introduces projections with the following additions to the API: * `khepri_projection:new/3` - Create a projection record. * `khepri_projection:name/1` - Retrieve the name (the ETS table name) of the projection. (Necessary because the projection record is opaque). * `khepri:register_projection/4` - Register a projection within the store against a given path pattern. As well as the internal implementation in `khepri_machine` and the associated documentation.
With this change, if any projections exist, they are listed in the output of `khepri:info/2`.
the-mikedavis
force-pushed
the
md-projections
branch
from
8cd3925
to
64df3a4
Compare
Projections are a system for creating replicated ETS caches available on all members of a Khepri cluster. A new function
khepri_projection:new/3creates a projection resource:The projection resource may then be registered to a pattern of nodes with
khepri:register_projection/4:Once registered, the projection's ETS table is created and then filled with any existing records matching the pattern. Paths and data are passed through the
ProjectionFunto create records which are then stored in the table. Any future changes to records which match thePathPatternare also applied to the projection table.The projection table is a named ETS table which may be queried directly with
ets:Projections should be used to maximize query throughput and/or minimize query latency at the cost of consistency and memory consumption. Projections have the same consistency guarantees as queries tried with the
#{favor => low_latency}option. Data stored in the projection table is at least partially duplicated between the store and the projection table, so increased memory consumption is expected for projections with path patterns matching many nodes.