Heavy LOH allocations in MessageCenter.TryForwardRequest

[Tragetaschen]

I'm currently analyzing a production workload that trips over from LOH allocations at a certain point. This seems to correlate heavily with the number of grains in an individual silo, ~10,000 being the "magic number". The deployment has grown to 80 silo instances hosting >500,000 grains.

The extracted memory allocation profiling pretty much entirely points to that one method as the sole cause for everything:

It looks before inlining that it's actually Message.AddToCacheInvalidationHeader, which seems to employ a pretty bad List<T> pattern:

• new List (size 0, capacity 4)
• AddRange (size according to argument, capacity == size)
• Add (size + 1, capacity *= 2)

which allocates thrice the needed amount and wastes half of the list's capacity. It could create the new list with the right capacity already.

Is this indicative of a different issue in our setup? Is there a way to avoid / reduce the allocations beyond just setting the right capacity from the get go?

[ReubenBond]

Do you know why you're seeing so many cache invalidations to begin with? Could you tell me more about your system?

[Tragetaschen]

The system hosts one customer session per Grain. There is no communication between Grains. A frontend service accepts connections from the internet and routes those into the Orleans cluster. Grains without activity in the session call DeactivateOnIdle after cleaning up the internal state. There is no persistence.

It looks like this particular trace was done while a rollout was unstable. A new version would gradually start, but run into a Kubernetes namespace limit and then roll back half way through. The new silos would typically get pretty much all the new traffic, so rolling those back would "kill" the most active sessions (the grain would probably be migrated to an existing silo, but without persistence it would be empty). This sounds like it could explain the cache invalidation...?