Consistency timeout when starting multiple dynamic commanded applications

[ghost]

I found this issue when trying to implement dynamic commanded applications for our project. It seems like when there is more than one dynamic app running, apps that are started later will experience consistency timeout for almost every command, while the first app would remain stable.
And the reason for consistency timeout seems to because, by the time consistency guarantee middleware tries to check if events are handled by all subscriptions, the info hasn't been updated in the streams_table yet while it should have been handled.

This is a sample app that replicated the issue I found: https://github.com/hwenyi/dynamic_cmd

[slashdotdash]

I've finally been able to take a look at this and have tracked down the root cause. It is because the event store schema is being shared between the dynamic Commanded applications. This causes a problem because the event store subscriptions only allow a single named subscriber.

When you start an event handler or process manager multiple times, once per dynamic application, they will each attempt to subscribe to the event store using the same subscription name. The event store ensures only one named subscription will be connected and can process events. Thus any handlers started for the second, third, fourth (etc.) dynamic Commanded application will not be able to subscribe to the event store.

There are two ways to resolve this issue:

1. Use a separate event store database or Postgres schema for each dynamic application.

   This will work because the identically named event store subscriptions will be tracked in the subscriptions table within the separate schemas. I have sent you a pull request with this fix in place.

2. Use the same event store database for all dynamic applications but define the event handler and process manager names at runtime so that they are uniquely named (e.g. by adding a dynamic suffix to their names).

   children = [
     {DynamicApplication, name: app_name},
     {DynamicEventHandler, application: app_name, name: "DynamicEventHandler" <> to_string(app_name)},
     {DynamicProcessManager, application: app_name, name: "DyanmicProcessManager" <> to_string(app_name)}
   ]
   
   Supervisor.start_link(children, strategy: :one_for_one)

   In the above example the event handler and PM processes are given a unique name with a suffix based upon the app name. This means each handler has a unique subscription name and can be tracked within a single subscriptions table by the Postgres event store.

   Support for runtime event handler and process manager names was recently added (Add init/1 callback function to event handlers and process managers #393) and is included in the v1.2.0 release.

   One issue you will need to address with this approach is that the event handlers and process managers will receive events from all applications due to the shared event store. You will likely need to filter and only process events associated with the application the handler was started for. Passing state, such as the tenant, to the handlers might help with this filtering (Support custom state for event handlers #400).

Using a separate event store database/schema would would be the preferred approach and is the intended usage of Commanded dynamic applications.