Multitenant implementation

[brettwinters]

I'm looking at migrating my event sourced / multitenant application to Eventuous. I've read through the docs and just want to check if the following is feasible before I dive in:

Each tenant has their own EventStore and database for projections/query models. There are also a few shared projections which are stored in a shared database - this projection would receive events from all tenants and is used for things like listing the tenants etc

My idea is this:

1. Resolve the tenant information on each request (connection strings, etc)

2. Change builder.Services.AddSingleton(streamNameMap); to builder.Services.AddScoped(streamNameMap);. Not sure if the CommandService should be / is already scoped to the request.

In order for projections to work, can the 'global sequence number' or event number be globally unique/sequential- like using ticks or some other incrementing mechanism?

Regards

Brett

[alexeyzimarev]

2. Change builder.Services.AddSingleton(streamNameMap); to builder.Services.AddScoped(streamNameMap);. Not sure if the CommandService should be / is already scoped to the request.

For what service you want to change from singleton to scoped?

[diegosasw]

I have no answer for what you're trying to achieve, but if it helps, I am also implementing multi-tenancy with Eventuous but with a different approach where they all share the same EventStoreDb and the same MongoDb (for projections)

The tenantId goes in the JWT for each request and I use that to determine the stream name where to write. So, basically I use StreamNameMap to append a tenant Id to the stream.
That way, during projections, when I retrieve the stream ID I also know the tenantId and I can append the tenantId to the document Id or index it and use it to filter things for that tenant on each request.

streamNameMap.Register<BrandId>(brandId => IdFormatter.ToStreamName("brand", brandId.Value, brandId.TenantId));
services.AddCommandService<BrandCommandService, Brand>();

then in the projections you can choose whichever ID generator mechanism. In my case something like this

var brandId = IdFormatter.ToDocumentId(ctx.Stream);

public static string ToDocumentId(
    string streamName, 
    char streamNameSeparator = '-', 
    char documentSeparator = '-',
    char hiddenPrefixSeparator = '#')
{
    var curedStreamName = streamName.Split(hiddenPrefixSeparator).Last();
    var chunks = curedStreamName.Split(streamNameSeparator);
    var relevantChunks = chunks.Skip(1);
    var result = string.Join(documentSeparator, relevantChunks);
    return result;
}

If I had to go with physical DB/EventStore separation because logical separation was not an option, I would also separate the application instances altogether and have each one of them pointing to the desired databases, and have the tenant discrimination at some api gateway, load balancer or similar to properly redirect traffic to the desired one as per request's tenant info.

There are also a few shared projections which are stored in a shared database - this projection would receive events from all tenants and is used for things like listing the tenants etc

I guess that's why you ask about globally unique ID. But in the projection you could always tweak the document Ids to ensure they're globally unique regardless of which stream they come from and just by knowing which tenant they belong to.

Probably not the answer you're looking for (sorry!) and not really related to Eventuous, but if I had a system that required combining info from different tenants, I would consider logical separation instead of physical separation. One of the good things of streams is that you have that logical separation already for free, as long as the tenant Id is part of the stream name.

[alexeyzimarev]

Yeah, but @brettwinters want to have a separate cluster for each tenant. For the shared read model I see a need to forward events somewhere: a separate ES cluster/instance or a log-based broker. It's because of the burden of managing many subscriptions on the shared read model subscriber side, I don't think it will be sustainable. I believe each tenant would need a subscription that knows nothing about the shared read model projector, just shovels event to a shared place to which one subscription for the shared read model will subscribe.

Depending on the number of tenants and data isolation requirements, @diegosasw's approach is one solution. I know many companies doing it like this.

If you have to have a cluster per tenant, you can consider tenant-based rollout instead. Your application compute would hardly be greater than the infrastructure required per tenant, especially if you use serverless. If you deploy tenant-based, you don't need the complexity of splitting the calls inside the shared application workload. API calls can be effectively routed per tenant by an API gateway using the tenant id claim in the token.

[brettwinters]

Thanks for your quick replies and ideas guys. Yeah, I started using @diegosasw's approach, but then some clients wanted full EventStore isolation for commercial reasons.

I've implemented my approach now using a modified version of Cirqus - in this case I think all I needed to do was change the global event sequence number to datetime ticks and change the lifetime of the eventstore, commandprocessor and projectors, but it's getting heavily modified now along with a lot of other changes and wanted to go to something more supported and a more tested features. ATM I just keep the shared projector running as a singleton and the create the tenant scoped EventStore/Projectors once a TenantInfo object containing things like the ConnectionString, etc is resolved early on in the middleware pipeline.

Having separate tenant-based application is the next level up, but added a lot of operational complexity. Not sure I'm ready to go that way yet since there is not just one microservices and some are tenant based, others shared, etc.

Anyway, I can see that you don't support out-of-the-box, but might be doable depending how pluggable Eventuous is - I might play around with it a little later when I have some time.

One question : It's not a dealbreaker if the event sequence numbers are not perfectly sequential i.e. 1, 2, 15, etc and the projectors don't care either?

[alexeyzimarev]

Ok, let's consider your case where you want to have a single codebase and support multi tenancy inside it.

First, I don't really understand the need of the global sequence number. Do you mean you want to have all the events ordered even across tenants? Why is that? As tenants are completely independent, what problem are you aiming to solve with the global unique sequence number? Btw no, ticks won't work. Ticks aren't unique, and aren't reliable as the machine clock is not reliable.

Second, I see issues with resolving dependencies based on tenant. The scoped lifecycle limits the service lifetime to. single request, and you can definitely register command services as scoped dependencies. It will work well for the API. But, a command service depends on the store, I am not exactly sure how the store will figure out the connection string it needs to use?

One way to enable this would be for the API endpoint to create an async local context and put the tenant information there. You'd need to have a wrapper around IEventStore that will hold a collection of actual stores or connection strings for all the tenants. If you use the async local context, you can keep all the dependencies registered as singletons. Another option is also to use the tenant context and propagate it downstream as a scoped dependency. In that case, all the dependencies need to be registered as scoped.

So that is doable, but I have doubts about subscriptions. Let's say you have 20 tenants and you need to project from their event stores to their read model databases. If you don't want to have one service per tenant, you'd need to have a single service for all the subscriptions, and you will need to start 20 catch-up subscriptions. Here using scoped dependencies won't help as you literally need to have 20 subscriptions because they subscribe to all the event stores. It is not a problem technically, but from the ops point of view it will be a risky service to monitor. Yes, all the subs will run independently, but it would require substantial compute and IO resources to run efficiently. You can of course distribute the load by assigning subsets of tenants to individual instances of that service, but you'd need to have to implement it depending on the hosting environment. For example, if you use stateful sets in Kubernetes, you can divide the list of tenants by the number of pods in the set and use the pod number as a "partition" number.

Maybe an easier way to do it is to move to the ops space and extend the tenant deployment (you need automation there anyway) with adding a connector workload to each tenant deployment. It would probably depend on the "generic HTTP" sink feature, which I am now working on (the New Sinks project). Connector is a drop-in component, you can consider it "infrastructure", not custom code. Your projector would then implement sending updates to a tenant-specific database. Projectors could then be deployed in any environment, including serverless, as it will be an on-demand push of events.

[brettwinters]

First, I don't really understand the need of the global sequence number.

True for the tenant projections, but won't the shared projections need to know the absolute checkpoint?

Yeah, know about ticks. I did something like this to prevent collisions:

public static long UtcNowTicks
{
	get
	{
		long original, newValue;
		do
		{
			original = _lastTimeStamp;
			long now = DateTime.UtcNow.Ticks;
			newValue = Math.Max(now, original + 1);
		} while (Interlocked.CompareExchange(ref _lastTimeStamp, newValue, original) != original);

		return newValue;
	}
}

But if I don't need then even better. This was just a hack to get it working.

I am not exactly sure how the store will figure out the connection string it needs to use?

I'm thinking something like this: (just noticed that you answered already in next line)

builder.Services
    .AddScoped(p => {
         var tenant = p.GetRequiredService<ITenantInfo>(); //from tenant resolver in the middleware pipeline
         return new EventStoreClient(new EventStoreClientSettings.Create(tenant.connectionString));
    })
    .AddAggregateStore<EsDbEventStore>();

About the subscriptions: I'm currently doing this (note my app / user count is very small, so you'll probably say its not scalable, etc):

1. Application Start. Shared projections initialise and stay running (as singletons with their own db), but since there are no requests (tenant unknown) nothing happens.

BTW I have a full catchup command if I need to rebuild the view models (which is mostly needed for shared view models) which I think was your question - on startup I get the tenant connection strings from an external Vault service and then loop through each tenant -> IServiceProvider.CreateScope() and then (optionally purge) and stream events to the subscriptions. this may take a minute or so).

2. Request comes in, tenant resolved by the middleware and makes ITenantInfo available in DI container
3. Then Startup the EventStore/CommandProcessor/Subscriptions for the tenant and do any catchups based on EventStore position (I think Eventuous does this automatically)
4. Then processes the command or query
5. If events are emitted by the CommandProcessor then they are dispatched to shared subscriptions + tenant specific subscriptions which process those events and update view models.

[alexeyzimarev]

I think subscriptions don't need to be scoped, unless you use scoped dependency like DbContext from EF Core. For the latter case I was using another solution (create scope in the handler).

For subscriptions, if you really want to host multiple subscriptions in one service, and each one of them connects to its own store, there is a way of adding dependencies when registering the subscription. So, it might be its own ESDB client, and its own target store (MongoDB database, SQL connection factory, etc).

It's done using something called ParameterMap and extensions AddParameterMap for the subscription builder. It's used, for example, to add a custom checkpoint store per subscription using UseCheckpointStore extension.

You can write something like:

services.AddSubscription<AllStreamSubscription, AllStreamSubscriptionOptions>(
    $"Projections-{tenantId}",
    builder => builder
        .UseCheckpointStore<MongoCheckpointStore>(_ => new SomeStore(tenantId))
        .AddParameterMap<EventStoreClient>(_ => new EventStoreClient(tenantConnectionString))
        .AddEventHandler<BookingStateProjection>()
        .AddEventHandler<MyBookingsProjection>()
        .WithPartitioningByStream(2)
);

The AddParameterMap<T> extension with dependency resolution factory is not in main yet (see #214)

[brettwinters]

Oh, I think I got it. Is this correct:

(1) Command processor is scoped for each request. The event store has it's own event counter. No need a globally unique sequence number.

(2) The checkpoint store is shared for all tenants and would keep a record of the last processed event for each tenant using a compound key with the tenantId

(3) The subscription services is shared service for all tenants. Receives events (prefixed or somehow knows the tenantId). The AddParameterMap feature would work like a handler scoped approach? resolves the ConnectionString (via my own resolver) and updates the view models...

[alexeyzimarev]

I think you changed the requirement a bit here.

You mentioned dedicated event stores per tenant, and it is currently not supported within a single app. Registering command services as scoped kind of makes little sense as they only get the event store injected, and you can't resolve it by tenant anyway.

If you use one event store for all the tenants and separate them by, say, stream prefix - there's no issue with that.

Projecting to different databases is not a problem. Subscriptions must be singletons, but what you do inside is up to you. You can inject the service provider and use it as a service locator to resolve dependencies per event. I would not recommend running one subscription per tenant as it will need to maintain multiple subscribers. It's easier to listen to $all and project events based on the context. You don't need to use the parameter map in this case, as you'd have to resolve the projection target per event. Parameter map would allow you to maintain one subscription per tenant, but it's only relevant if you have multiple event stores you subscribe to.

[brettwinters]

"Each tenant has their own EventStore and database for projections/query models" here

If changing the scopes of the command handler/EventStore is not possible, I think the dedicated stores per tenant is solvable, say using something like Autofac.Multitenant, but issue is still how to handle any shared subscriber/view models (shared between all tenants - for example, a list of all tenants. In this case the ViewModel would be updated by subscribing to the TenantRenamed event, as an example)

I see the problem statement as follows: if shared subscriber is getting events from multiple stores then the checkpoint can't be used because sequence numbers of the event stores will overlap.

Example:
Let's say ES1 at position : 10001, ES2 at 222. If checkpoint for a shared subscription at 100001 and then receives event 223 from ES2 it will be ignored because it's already ahead, right?

Could AddParameterMap solve this issue?

[alexeyzimarev]

I don't think it's about scopes of the command service. I don't really understand how registering command services as scoped dependencies will help resolving different instances of the stores.

When you mentioned things like the list of all tenants, I start to believe you'd need to have a "top-level" store, which will keep the consolidated information. It could be things like subscriptions, invoices, contacts, etc. It is your stuff, not tenant's stuff.

[brettwinters]

I don't think it's about scopes of the command service. I don't really understand how registering command services as scoped dependencies will help resolving different instances of the stores.

Currently, I resolve the tenant and get the ConnectionString on each request. So I was thinking that the command service needs to be instantiated then to get the AggregateStore, etc.
Let me investigate further how the IEventStore, AggregateStore and CommandService work and get back to you with something more specific. I will try with one of my single-tenant services first to learn how it works.

When you mentioned things like the list of all tenants, I start to believe you'd need to have a "top-level" store, which will keep the consolidated information. It could be things like subscriptions, invoices, contacts, etc. It is your stuff, not tenant's stuff.

Yes, that's possible. I've done that in the past. Basically in the command handler / CommandApi you'd do two things. First pass the command to the ICommandService for usual processing, second create /update DB record in your own store. It's not as elegant but works.

[alexeyzimarev]

Basically, ASP.NET Core DI has never been designed to support multi tenancy where one dependency service can have tenant-based implementation. You mentioned named Autofac registrations, that worked.

The closest thing in current .NET is how they advise to use IHttpClientFactory and get a named client instance from it. So, you take factory as a dependency. For event handlers it's very easy to do. For command services - not that much, at least without changing the code.

[alexeyzimarev]

Another thing is that command services do some (relatively) heavy lifting on instantiation for the sake of fast message handling. So, you would really prefer them as singletons. Another question is how to get the scoped store into it.

[alexeyzimarev]

Check the PR, it should address the command handling part

[alexeyzimarev]

However, I am not exactly happy about this design. It seems I keep adding more transformations to the command handler instead of using filters. Need to think more about it.

[alexeyzimarev]

So, I made drastic API changes in both aggregate and functional services to make the API more lean. Now it should be possible to add more options to how commands are being handled, as a method chain. This will not longer be a concern:

However, I am not exactly happy about this design. It seems I keep adding more transformations to the command handler instead of using filters. Need to think more about it.

[alexeyzimarev]

It's not exactly filters, but a handler builder instead. The API is close to creating a pipe though.

[alexeyzimarev]

Ok, the 0.15.0-beta.4 contains the new API. The old API is usable, but marked obsolete. Let me know if it actually help with multi-tenancy, I expect it does.

[brettwinters]

Let me take a look. I got sidetracked then bogged down in another issue.