Releases · rabbitmq/khepri

28 Mar 14:00

dumbbell

v0.13.0

076926a

Khepri 0.13.0 Latest

Latest

What's new in Khepri 0.13.0

Preliminary support for Erlang/OTP 27

With the upgrade of Horus from 0.2.4 to 0.2.5, Khepri gains preliminary support for Erlang/OTP 27. We say preliminary because Erlang/OTP 27 has not been released yet, so we can't guarantee at that point that it will be fully compatible with the final version.

See #254.

Other changes

Fixed a bug where projections could be out-of-sync with the actual database tree content because of the way they were initialized on Ra server recovery (#255).
Made the #khepri_machine{} record private and opaque (#252, #253).

Download

Khepri 0.13.0 is available from Hex.pm: https://hex.pm/packages/khepri/0.13.0

Upgrade

Using Rebar:

Update your rebar.config:

%% In rebar.config
{deps, [{khepri, "0.13.0"}]}.

Run rebar3 upgrade khepri.

Using Erlang.mk:

Update your Makefile:

%% In your Makefile
dep_khepri = hex 0.13.0

Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.13.0 is available on Hex.pm.

Assets 2

21 Feb 10:29

dumbbell

v0.12.1

c49aeba

Khepri 0.12.1

What's new in Khepri 0.12.1

Handling of unknown commands

Now, if a Khepri state machine receives an unknown command, it will throw an exception. This will be useful in the future in case of an upgrade of the state machine code if new commands are introduced. If the new commands are submitted before the Ra cluster upgraded the state machine version, then the caller will be notified the command couldn't be applied.

See #251.

Download

Khepri 0.12.1 is available from Hex.pm: https://hex.pm/packages/khepri/0.12.1

Upgrade

Using Rebar:

Update your rebar.config:

%% In rebar.config
{deps, [{khepri, "0.12.1"}]}.

Run rebar3 upgrade khepri.

Using Erlang.mk:

Update your Makefile:

%% In your Makefile
dep_khepri = hex 0.12.1

Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.12.1 is available on Hex.pm.

Assets 2

12 Feb 12:09

dumbbell

v0.12.0

35a9f5b

Khepri 0.12.0

What's new in Khepri 0.12.0

Ra 2.9.1

This new release of Ra comes with an important bug fix to the consistency of the list of members/nodes returned by khepri_cluster:members/{0,1,2} and khepri_cluster:nodes/{0,1,2}. Before this change, it was possible that the returned list was out-of-date and stale.

See rabbitmq/ra#417 and #249.

Other changes

Fixed unregistering of non-existent projections (#248).

Download

Khepri 0.12.0 is available from Hex.pm: https://hex.pm/packages/khepri/0.12.0

Upgrade

Using Rebar:

Update your rebar.config:

%% In rebar.config
{deps, [{khepri, "0.12.0"}]}.

Run rebar3 upgrade khepri.

Using Erlang.mk:

Update your Makefile:

%% In your Makefile
dep_khepri = hex 0.12.0

Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.12.0 is available on Hex.pm.

Assets 2

30 Jan 08:50

dumbbell

v0.11.0

e85821b

Khepri 0.11.0

What's new in Khepri 0.11.0

Return `{ok, _} | {error, _}` from functions returning cluster members

Always returning a list, hiding any error behind an empty list, proved to be a bad design, as it made it way too easy to ignore or miss errors and act upon them.

This kind of return value was also inconsistent with the rest of the API.

The members list is now wrapped into an {ok, Members} tuple. Likewise for nodes lists. This forces the caller to be aware of errors and handle them. The afftected APIs are:

khepri_cluster:members/{1,2}
khepri_cluster:locally_known_members/{1,2}
khepri_cluster:nodes/{1,2}
khepri_cluster:locally_known_nodes/{1,2}

Here is an example of the change to make to your code:

Up-to Khepri 0.10.x:

lists:foreach(
    fun(Node) ->
        do_something_with_node(Node)
    end,
    khepri_cluster:nodes(StoreId)).

Starting from Khepri 0.11.0:

case khepri_cluster:nodes(StoreId) of
    {ok, Nodes} ->
        lists:foreach(
            fun(Node) ->
                do_something_with_node(Node)
            end,
            Nodes);
    {error, _Reason} = Error ->
        handle_error(Error)
end.

While working on this:

khepri_cluster:members/0, khepri_cluster:locally_known_members/0, khepri_cluster:nodes/0 and khepri_cluster:locally_known_nodes/0 were added to work on the default store. This is inline with all other API functions.
Type specifications and documentation was added for all these functions, existing and new.

See #244.

Speed up members/nodes local query

Ra now provides a new ra_leaderboard:lookup_members/1 function to query members locally that is faster that a call to the local Ra server. Khepri started to use it and, if it returns an error, it will fall back to the previous Ra server query.

See #240.

Other changes

Use monotonic times to mesaure a duration, not system time (#245).
Do not enforce the favor option when querying the "keep while" conditions or the projections states (#246).
Ra was bumped from 2.7.0 to 2.7.3 (#241).

Download

Khepri 0.11.0 is available from Hex.pm: https://hex.pm/packages/khepri/0.11.0

Upgrade

Using Rebar:

Update your rebar.config:

%% In rebar.config
{deps, [{khepri, "0.11.0"}]}.

Run rebar3 upgrade khepri.

Using Erlang.mk:

Update your Makefile:

%% In your Makefile
dep_khepri = hex 0.11.0

Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.11.0 is available on Hex.pm.

Assets 2

12 Dec 09:32

dumbbell

v0.10.1

e4d2529

Khepri 0.10.1

What's new in Khepri 0.10.1

This release fixes the list of files published to Hex.pm: the LICENSE-* files were missing.

See #237.

Download

Khepri 0.10.1 is available from Hex.pm: https://hex.pm/packages/khepri/0.10.1

Upgrade

Using Rebar:

Update your rebar.config:

%% In rebar.config
{deps, [{khepri, "0.10.1"}]}.

Run rebar3 upgrade khepri.

Using Erlang.mk:

Update your Makefile:

%% In your Makefile
dep_khepri = hex 0.10.1

Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.10.1 is available on Hex.pm.

Assets 2

11 Dec 17:29

dumbbell

v0.10.0

0b8a99d

Khepri 0.10.0

What's new in Khepri 0.10.0

This release brings a single API change around the async operations.

Unify `khepri:handle_async_ret/2` return values

The new khepri:handle_async_ret/2 introduced in Khepri 0.9.0 is improved in this new release: it converts the "raw" return values contained in the ra_event to a format similar to all other Khepri sync commands. This allows the function to handle specific cases like exceptions and aborted transactions.

Here is an example of the change to make to your code:

Up-to Khepri 0.9.x:

receive
    {ra_event, _,
     {applied, [{CorrelationId, {ok, _} = Ret}]}} = AsyncRet ->
        %% Call `handle_async_ret/2' for every entries;
        %% here only one:
        ok = khepri:handle_async_ret(StoreId, AsyncRet),

        %% Do something with each `Ret'.
        Ret
end

Starting from Khepri 0.10.0:

receive
    {ra_event, _, _} = AsyncRet ->
        %% Call `handle_async_ret/2' once and get a list of correlation
        %% ID/return value pairs.
        [{CorrelationId, Ret}] = khepri:handle_async_ret(StoreId, AsyncRet),

        %% Do something with each `Ret'.
        Ret
end

See #234.

Download

Khepri 0.10.0 is available from Hex.pm: https://hex.pm/packages/khepri/0.10.0

Upgrade

Using Rebar:

Update your rebar.config:

%% In rebar.config
{deps, [{khepri, "0.10.0"}]}.

Run rebar3 upgrade khepri.

Using Erlang.mk:

Update your Makefile:

%% In your Makefile
dep_khepri = hex 0.10.0

Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.10.0 is available on Hex.pm.

Assets 2

03 Nov 10:20

dumbbell

v0.9.0

90b8c1c

Khepri 0.9.0

What's new in Khepri 0.9.0

This release focuses on a couple improvements and bug fixes to async commands.

Improvements to async commands

Correctly use the cached leader ID if it is present to send the command (#228).
Replace khepri:wait_for_async_ret/{1,2} with khepri:handle_async_ret/{1,2} (#229). The previous API didn't match the underlying Ra API. It's better for the caller to receive the message(s) from the Ra server, then call khepri:handle_async_ret/{1,2} to let Khepri do internal things such as updating the cached leader ID.

Other changes

Expose the ?IS_KHEPRI_STORE_ID/1 macro (#231).
Ra was bumped from 2.6.3 to 2.7.0 (#232).

Download

Khepri 0.9.0 is available from Hex.pm: https://hex.pm/packages/khepri/0.9.0

Upgrade

Using Rebar:

Update your rebar.config:

%% In rebar.config
{deps, [{khepri, "0.9.0"}]}.

Run rebar3 upgrade khepri.

Using Erlang.mk:

Update your Makefile:

%% In your Makefile
dep_khepri = hex 0.9.0

Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.9.0 is available on Hex.pm.

Assets 2

30 Aug 14:17

dumbbell

v0.8.0

952a529

Khepri 0.8.0

What's new in Khepri 0.8.0

This release focuses on several important bug fixes and improvements to projections.

Many improvements to projections

Projections are a new mechanism introduced in Khepri 0.6.0 to allow fast queries with a low latency.

Here are five pull requests related to projections:

Add a khepri:unregister_projection() function to unregister projections (#220).
Make "copy"-style projections more efficient and straightforward to use (#204).
Handle errors from ETS gracefullly (#205).
Allow to pass options to Horus (#209).

Other changes

New khepri:is_empty/1 public function (#210).
khepri_cluster:join/3 is now idempotent (#219).
Fix some corner cases in khepri_cluster:join/3 (#225).
Handle missing default applications when determining the list of modules to skip when extracting a transaction function (#215).
Handle Ra server shutdown gracefully (#211, #223).
Improve logging for member queries (#208, #216).
Fix documentation of put() (#206).
Ra was bumped from 2.5.1 to 2.6.3 (#224).
Horus was bumped from 0.2.3 to 0.2.4 (#224).

Download

Khepri 0.8.0 is available from Hex.pm: https://hex.pm/packages/khepri/0.8.0

Upgrade

Using Rebar:

Update your rebar.config:

%% In rebar.config
{deps, [{khepri, "0.8.0"}]}.

Run rebar3 upgrade khepri.

Using Erlang.mk:

Update your Makefile:

%% In your Makefile
dep_khepri = hex 0.8.0

Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.8.0 is available on Hex.pm.

Assets 2

17 Apr 09:15

dumbbell

v0.7.0

e5b8753

Khepri 0.7.0

What's new in Khepri 0.7.0

This release focuses on several important bug fixes and improvements to projections.

We promoted this release from Alpha to Beta as we believe the API is a lot more stable now. We are not ready to commit to this API yet though and may introduce more breaking changes if the need arises.

Many improvements to projections

Projections are a new mechanism introduced in Khepri 0.6.0 to allow fast queries with a low latency.

Here are five pull requests related to projects:

The deletion of an entire subtree is reflected in projections (#178).
Projections' ETS tables are correctly recreated when a Ra server is restarted and enters the recovered state (#182, #193)).
Query speed improvements (#188).
Reduce memory footprint (#190).

Creation of the Horus library

The previous khepri_fun module was extracted from Khepri and the Horus library was created out of it. It makes this code easier to maintain and test, and it can be reused outside of Khepri.

See #192.

Split of the `khepri_machine` code into several modules

The khepri_machine module was really large because it hosted the entire implementation of the internal tree data structure. This module was refactored and split into multiple modules to improve the readability of the code and its maintenance.

See #180, #181, #185, #186, #187, #189 and #194.

Other changes

Documentation improvements (#173).
Supervision tree fixes (#174).
New khepri_cluster:is_store_running/1 helper (#175).
Work around Rebar's use of cth_readable parse_transform (#176).
Support for new Erlang/OTP 26 assembler instructions (#183).
Ra was bumped from 2.4.0 to 2.5.1 (#191, #199).

Download

Khepri 0.7.0 is available from Hex.pm: https://hex.pm/packages/khepri/0.7.0

Upgrade

Using Rebar:

Update your rebar.config:

%% In rebar.config
{deps, [{khepri, "0.7.0"}]}.

Run rebar3 upgrade khepri.

Using Erlang.mk:

Update your Makefile:

%% In your Makefile
dep_khepri = hex 0.7.0

Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.7.0 is available on Hex.pm.

Assets 2

14 Nov 16:09

dumbbell

v0.6.0

7c0a4e5

Khepri 0.6.0

What's new in Khepri 0.6.0

This release focuses on several important new features.

Very fast queries thanks to "projections"

Projections are a new mechanism to allow fast queries with a low latency.

Projections use ETS tables to work as a cache. The user is in control of the way data is stored in these ETS tables:

He selects what is to be cached using path patterns.
He provides an anonymous function to transform the data payload to a record ready to be stored in ETS.

This obviously comes at a cost. The cache being implemented as ETS tables means data duplication between the store itself and the ETS tables and an increase in memory footprint. Also, queries are therefore eventually consistent (because of the time it may take to reflect a store change in the cache).

Here is a copy of the exemple in the pull request:

You need to declare the projection first:

%% `ProjectionName' will be the name of the ETS table.
ProjectionName = wood_stocks,

%% `ProjectionFun' defines how a data payload is transformed into an ETS
%% record.
ProjectionFun = fun([stock, wood, Kind], Stock) -> {Kind, Stock} end,

%% This line creates an opaque term from the variables above.
Options = #{type => set, read_concurrency => true},
Projection = khepri_projection:new(ProjectionName, ProjectionFun, Options),

%% You can finally associate the created projection with a path pattern.
StoreId = stock,
PathPattern = "/:stock/:wood/*",
RegisterOptions = #{},
khepri:register_projection(StoreId, PathPattern, Projection, RegisterOptions).

The ETS table named wood_stocks will be created and filled with the existing tree nodes matching the path pattern, after they are transformed using the given anonymous function.

You can start to query the ETS table for cached values:

%% Imagine you write a new value in the store, whether it is before or
%% after creating the projection.
khepri:put(StoreId, "/:stock/:wood/Oak", 100),

%% After the projection was defined, you can query the ETS table:
100 = ets:lookup_element(ProjectionName, <<"Oak">>, 2).

See #135 for more information.

Import and export of a store

This release introduces an import/export mechanism.

This feature relies on import/export backend modules. These modules follow the same API as the "backup and restore" Mnesia API, therefore it is possible to re-use existing backup modules for Mnesia.

Unlike Mnesia however, it is possible to export a part of the store by specifying a path pattern to select tree nodes to export.

During import, everything is written to the store. Note that other existing tree nodes are left untouched: the store is not emptied before an import.

Khepri 0.6.0 comes with an import/export backend module called khepri_export_erlang which uses a plaintext file where tree nodes are formatted as Erlang terms. The benefit is it doesn't add any dependencies to the library.

Here is an example:

Export the wood stocks from an existing store:

ok = khepri:export(
   StoreId,
   "/:stock/:wood/**",
   khepri_export_erlang,
   "export-wood-stock-1.erl").

export-wood-stock-1.erl contains:

{put, [stock, wood, <<"Oak">>], {p_data, 100}, #{}}.
{put, [stock, wood, <<"Maple">>], {p_data, 230}, #{}}.

Import the wood stocks back:

ok = khepri:export(
   StoreId,
   khepri_export_erlang,
   "export-wood-stock-1.erl").

See #157 for more information.

New `maps:fold/3`-like functions

To make it easier to work on many tree nodes, Khepri 0.6.0 brings the common iterative functions we are used to use in lists and maps modules:

khepri:fold/5
khepri:foreach/4
khepri:map/4
khepri:filter/4

They will all loop on tree nodes matching the given path pattern and run the given anonymous function for each one of them. Except for khepri:foreach/4, the return value of this anonymous function will be used to build a new Erlang term return by the khepri:fold/5 and friends.

Example:

Fun = fun
          (Path, #{data := Stock}, Acc) ->
              Wood = lists:last(Path),
              Acc#{Wood => Stock};
          (_Path, _NodeProps, Acc) ->
              Acc
      end,

#{<<"Oak">> := 100},
 {<<"Maple">> := 230}} = khepri:fold(StoreId, "/:stock/:wood/*", Fun, #{}).

See #163 for more information.

Stored procedures as transaction functions

Transactions in Khepri can be expensive in terms of CPU and memory because of the need to extract the given anonymous function, verify what it does and create a standalone module out of it to be executed on all Ra nodes.

To "pay" this extraction cost once, it is now possible to use stored procedures as transaction functions. In other words, a caller needs to:

store the transaction anonymous function once in the store
call khepri:transaction/2 with the path to that stored procedure instead of passing the anonymous function directly

To make transactions more flexible and allow the use of more generic stored procedures, transaction functions can now take arguments.

See #150 for more information.

Other changes

Automatically reclaim useless tree nodes which were automatically created (#155).
Support empty binaries in string-based paths (#156).
Fix handling of khepri:delete_many("**") (#158).
Rename delete_payload() back to clear_payload().
Fix application definition when building Khepri with mix(1) (#146).
Add module_info/{0,1} functions to extracted stored procedures and transaction functions, as required by debugging tools (#148).
Fix several issues around function extraction: #149, #161, #165, #168, #169, #170.
Acknowledge trigger executions asynchronously to fix a race with store shutdown (#151).
Fix regression with the use of #if_has_data{} with node props (#154).
Implement ra_machine:overview/1 for khepri_machine (#159).
Reduce the size of the "put" Ra commands by merging two maps (#166).
Accept machine_config in a Ra server config map (#167).

Download

Khepri 0.6.0 is available from Hex.pm: https://hex.pm/packages/khepri/0.6.0

Upgrade

Using Rebar:

Update your rebar.config:

%% In rebar.config
{deps, [{khepri, "0.6.0"}]}.

Run rebar3 upgrade khepri.

Using Erlang.mk:

Update your Makefile:

%% In your Makefile
dep_khepri = hex 0.6.0

Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.6.0 is available on Hex.pm.

Assets 2

Releases: rabbitmq/khepri

Khepri 0.13.0

What's new in Khepri 0.13.0

Preliminary support for Erlang/OTP 27

Other changes

Download

Upgrade

Using Rebar:

Using Erlang.mk:

Documentation

Khepri 0.12.1

What's new in Khepri 0.12.1

Handling of unknown commands

Download

Upgrade

Using Rebar:

Using Erlang.mk:

Documentation

Khepri 0.12.0

What's new in Khepri 0.12.0

Ra 2.9.1

Other changes

Download

Upgrade

Using Rebar:

Using Erlang.mk:

Documentation

Khepri 0.11.0

What's new in Khepri 0.11.0

Return {ok, _} | {error, _} from functions returning cluster members

Speed up members/nodes local query

Other changes

Download

Upgrade

Using Rebar:

Using Erlang.mk:

Documentation

Khepri 0.10.1

What's new in Khepri 0.10.1

Download

Upgrade

Using Rebar:

Using Erlang.mk:

Documentation

Khepri 0.10.0

What's new in Khepri 0.10.0

Unify khepri:handle_async_ret/2 return values

Download

Upgrade

Using Rebar:

Using Erlang.mk:

Documentation

Khepri 0.9.0

What's new in Khepri 0.9.0

Improvements to async commands

Other changes

Download

Upgrade

Using Rebar:

Using Erlang.mk:

Documentation

Khepri 0.8.0

What's new in Khepri 0.8.0

Many improvements to projections

Other changes

Download

Upgrade

Using Rebar:

Using Erlang.mk:

Documentation

Khepri 0.7.0

What's new in Khepri 0.7.0

Many improvements to projections

Creation of the Horus library

Split of the khepri_machine code into several modules

Other changes

Download

Upgrade

Using Rebar:

Using Erlang.mk:

Return `{ok, _} | {error, _}` from functions returning cluster members

Unify `khepri:handle_async_ret/2` return values

Split of the `khepri_machine` code into several modules

New `maps:fold/3`-like functions