StreamProxy GrainService

[GeroL]

Hi,

consider following scenario:

• X Silos
• N streams to different exchanges with stock charts with different timeframes like 1 minute, 5 minutes, 1 month and so on.
  -> 1 Grain as stream producer which collects the stock data and publishes it as stream
• M actors to process these streams distributed evenly
  -> Y actors per Silo

The ratio between N:M ranges from 1:1 to 1:100000 or even more (no plans for limits yet)

I figured that when I use the standard pub-sub SMS provider each producer grain can send the data to a stream and the actor grains would receive it. Simple and it would work.

What my concern here is that each silo would receive Y copies of messages with many redundancies which would of course consume
unnecessarily much network bandwidth.

The solution I thought of, would be a proxy per silo for each stream.
A proxy implemented as GrainService subscribes to a producer Grain and distributes the stream in the silo to local grains.

This would essentially remove the redundancies on the network.

This would require these features:

• GrainService can subscribe to a stream producer
• GrainService can create a stream or can create a local Grain which creates a stream
  --> Workaround: GrainService passes the message to all local grains which process the stream
• Grains can subscribe to GrainService or local Grain

I have yet to figure out how a Grain can be created locally that has the same Id as in other silos.
Has the GrainService access to the name of the silo? I could use it to create a unique per-Silo Id.
Can a grain be forced to be created locally or fail if this were not possible?

We have also Pulsar in use, but I try to avoid this extra dependency as it would constrain other resources.

Other thoughts:
The Grains for the processing will be stateful. Some streams might not get data for weeks so the grains can
be activated and deactivated without time constraints.
The grains which have a timeframe <= 5 minutes would be deactivated actively.
Would this be good enough? Servers are VMs with at least 16 cores (AMD EPYC) and 64GB RAM

What do you think?

Thanks for some help

[GeroL]

I found this in the Discord chat. I will give it a try

https://discord.com/channels/333727978460676096/333727978460676096/1045424809036742666

@MarkNickeson — 24.11.2022 20:44
One solution I use for the above "observer scale out" is to have a proxy grain defined like so: [PreferLocalPlacement]public class LocalObserverProxy {}, and then use a grain key of siloID+targetGrainToObserveID. That way requests for the "observer proxy" on local silo resolve the proxy. And the proxy implementation itself is the only observer of the actual target, wherever it is in the cluster. In the model, if you have 1000 observers in a 10 node cluster, you get ~100 observers per node that are all local calls and most likely 10 proxy observers that have to do network I/O to keep up to date. The original target also has much less work to do notifying only 10 registered proxies instead of 1000.