Multiple Database support + Auto-snapshot saving and replay

[henry-hz]

Port this idea to commanded: https://github.com/work-capital/engine/blob/develop/lib/engine/storage/storage.ex
Let me suggest to have a "storage" abstraction here, and use https://github.com/slashdotdash/eventstore as Postgre dependency to connect to Postgree and https://github.com/exponentially/extreme to connect to 'GetEventStore'

[slashdotdash]

1. Create an Elixir behaviour to define the contract required for storage.
2. Implement a set of unit/integration tests against the behaviour that can be used to verify a given storage provider will work when used. The tests should be configured to run against each supported storage provider. This can be achieved using ExUnit.CaseTemplate.
3. Use optional: true to reference external dependencies that are supported inmix.exs (e.g. {:eventstore, "...", optional: true})

The individual storage adapters can be wrapped with Code.ensure_loaded to ensure that the code is only loaded when the consumer of Commanded has added the dependency.

if Code.ensure_loaded?(EventStore) do 
  # ... `eventstore` provider
end

if Code.ensure_loaded?(Extreme) do 
  # ... `extreme` provider
end

[slashdotdash]

Auto-snapshotting should be implemented as a separate feature from the storage abstraction. Since snapshotting can be run as a background activity. Completely separate from persisting events for an aggregate. This also ensures that there won't be increased latency on aggregate operations because a snapshot is also being recorded. See #27.

[henry-hz]

Ok, got it, good design :) I will continue with the database layer, with the contracts, and leave snapshots for the end. Let's avoid early optimizations. I also thought about a "persistence" layer that could receive aggregates and process managers data structures, replay them and send them back, using closures. So I call the closure callback injected the read events from db. The idea is to have a testable, abstract and unique place to persist and rehydrate states for all the application, @slashdotdash makes sense ?

[henry-hz]

hum... I am cooking:
https://github.com/work-capital/commanded/blob/master/lib/commanded/storage/behaviour.ex

[sharksdontfly]

Hello @henry-hz, @slashdotdash maybe this could be useful to you, I have a fork at https://github.com/sharksdontfly/commanded/tree/eventstore-behaviour which integrates geteventstore.com (via extreme). Event store usage in commanded is replaced by a behaviour. It has one limitation though, when using extreme event store ack'ing of events is not implemented. To run all tests for eventstore or extreme event store you have to execute MIX_ENV=test mix clean && COMMANDED_EVENT_STORE_ADAPTER=Commanded.EventStore.Adapters.ExtremeEventStore mix test or MIX_ENV=test mix clean && COMMANDED_EVENT_STORE_ADAPTER=Commanded.EventStore.Adapters.EventStoreEventStore mix test respectively. Testing extreme event store takes considerably longer because each time the event store is resetted a docker container is restarted. When using the extreme event store one has to configure extreme and additionally provide a value for :streams_prefix (see config/test.exs). So if you configure bankaccount here and an event for account ACC123 is written to the event store it will be written to the stream bankaccount-ACC123.

Additionally there is a module which one can use to customize event names, it works only when using the integrated Commanded.Serialization.JsonSerializer. E.g.:

defmodule Events do
  alias Commanded.Serialization.JsonSerializer.EventName

  defmodule MoneyTransferRequested do
    use EventName, "money-transfer-requested"

    defstruct [transfer_uuid: nil, debit_account: nil, credit_account: nil, amount: nil]
  end
end

[slashdotdash]

@sharksdontfly Wow. That's an awesome chunk work of work you've done to integrate Greg's event store.

Are you interested in submitting a pull request to merge the changes back in once you're done? That would help out @henry-hz too.

[henry-hz]

@sharksdontfly wow!
@slashdotdash , seems that we can discard the #42 PR.

[slashdotdash]

Done by #55.