Horizontal scaling plateau (10k msgs/min) & LogPersistenceMode bypass causing connection pool starvation

[olegkoutchine]

Hello Elsa team!

We are running a high-throughput load test (100,000 messages) through a custom Elsa 3 workflow. We've hit a strict compute/throughput ceiling of about 9-10 minutes, and scaling our Kubernetes pods horizontally is yielding zero performance improvements.

We suspect a combination of MassTransit/EF Core connection pool starvation, compounded by an issue where the C# engine is ignoring the logPersistenceMode flag and writing 1.1 million telemetry records anyway.

Would love your architectural advice on how to unblock this.

Architecture & Load Test Baseline
Infrastructure: Kubernetes, PostgreSQL backend (100 max connections), RabbitMQ via MassTransit.

Workflow: Simple custom workflow (Start -> Check Variables -> CustomSendEmail -> Finish).

Test: 100,000 messages pushed to RabbitMQ.

Time to process: ~10 minutes (approx. 166 workflows/sec).

The Scaling Problem
When we scale out our pods, the execution time remains exactly 10 minutes.

Run 1: 2 Worker Pods (Max Pool Size = 25) -> 10 minutes

Run 2: 5 Worker Pods (Max Pool Size = 10) -> 10 minutes

In both scenarios, we are maxing out at 50 total database connections across the cluster. We tuned MassTransit (ConcurrentMessageLimit=100, PrefetchCount=250), but because the engine is fighting for those database connections, adding more CPU/pods isn't speeding up the queue processing.

To reduce DB load, we attempted to disable telemetry logging on noisy nodes (Start, Wait, etc.) using logPersistenceMode: "Exclude", but looks like Elsa ignores it and we can see about 400000 records in ActivityExecutionRecords table and 728209 record in WorkflowExecutionLogRecords table.

Configuration Examples:

`// MassTransit / RabbitMQ Configuration
services.AddElsa(elsa =>
{
elsa.UseMassTransit(massTransit =>
{
massTransit.UseRabbitMq(rabbitMq =>
{
// Pushing limits to utilize CPU, but blocked by DB connections
rabbitMq.PrefetchCount = 250;
rabbitMq.ConcurrentMessageLimit = 100;
});
});

// DB Configuration (Throttled to prevent Postgres 100 connection crash)
elsa.UseDefaultHosting(hosting => 
{
    hosting.UseEfCore(ef => ef.UsePostgreSql("Host=...;Maximum Pool Size=10;"));
});

});`

Questions for the Team:

1. JSON Deserialization: Is there a supported way to format the JSON payload so the C# engine properly binds "Exclude" to the LogPersistenceMode enum when coming from the Studio UI's customProperties? Alternatively, is there a global way in Program.cs to entirely disable ActivityExecutionRecords generation to save DB overhead?

2. Database Connection Strategy: Since Elsa relies heavily on EF Core for workflow state, is deploying a connection pooler like PgBouncer the standard/recommended approach for scaling Elsa to 10+ pods?

3. Dispatcher Tuning: Are there other internal Elsa dispatcher/channel settings (outside of MassTransit concurrency) that we should be tuning to maximize throughput for short-lived workflows?

Thanks in advance for any insights!