How Orleans can be used in an IoT project?

[staviloglu]

Hi.

First of all thanks for the Orleans framework. I appreciate your efforts to make distributed application development a lot easier.

Orleans seems to be an exact match for IoT business where we need device twins on backend.

But unfortunately I can not seem to find a suitable example rather than the old GPS Tracker example which is working with an HTTP Frontend.

I was wondering if you could point me in the right direction and your thoughts on the following architectures that come to my mind.

I will use RabbitMQ and its MQTT Plugin to support MQTT connections from the devices on the field. They will periodically publish sensor data to given topics.

1- Each grain can create its own connection + channel and subscribe/publish to related queue on RabbitMQ. This does not seem to be a reasonable solution to me, because each device already makes connections if the device twins do that too we double the connection count which will lead to performance loss on RabbitMQ.

2- Each silo will have 2 connections (one for publish and one for subscribe) but each grain will create their own channels using the shared connections. This solution is ok for connection count but increase in channel count creates a lot less but almost the same effect on RabbitMQ. And how should I share the connections (1 stateless worker, IHostedService, GrainService?) Pooling connections come to my mind but first need to be sure.

3- I may create different processes for subscribing and publishing, each create their own connections and subscribe/publish to RabbitMQ. When a message arrived I can use GrainClient to send data to Orleans cluster. That seems totally fine to me, this way I can scale subscription process seperate from the silos. But I'm not sure how can I publish from grains to RabbitMQ which requires orleans to communicate with the publisher process. (gRPC etc? Orleans streams?)

4- I may have the subscriber process mentioned above but rather than a seperate publisher process, each grain can create their connection + channel and publish the data when required. And close the connection + cannel. Or share one connection per silo but seperate channel opened/closed in each grain.

5- Stream Providers? I have seen some documentation about pulling agents but I want to use RabbitMQ push based. I have seen Stream provider for RabbitMQ bu I think that does not solve the problem here. Am I wrong?

Tihs is might be too much to ask and time consuming to read. :(

Hope to see an answer from your side.

Thanks for your time.

[ReubenBond]

Hi @staviloglu,
Orleans is indeed a good match for IoT and has been put to use in many IoT applications, including in the backend of Azure IoT Digital Twins. We typically do not provide much input into overall application architecture - maybe we should, but it's time-intensive and ambiguous, being more suitable for discussion than Q&A. Regardless, please accept my apology for not responding sooner, as unfulfilling as this response may be.

I would generally use an IHostedService (eg BackgroundService) over a GrainService`. Using gRPC between services is a fine approach. Orleans streams are fine, too, and give you some benefits such as increased decoupling over gRPC (given gRPC is point-to-point and temporally coupled). Interfacing with RabbitMQ directly gives you more control, so perhaps that's suitable if your scenario doesn't fit into the virtual streams / firehose approach that Orleans Streams takes.

If anyone has opinions here, please feel free to chime in. I'll convert this to a discussion, since I think it's more suited for that

[ElderJames]

I am thinking of using HostedService as an MQTT Broker and accessing Grains directly using GrainFactory. instead of using ClusterClient.

I'm not sure at this point that this approach will allow the MQTT Broker to become a cluster. If not, is there another way to do clustering?Using Stateless Worker Grains?

[fuocor]

We've developed an IoT directed graph architecture that uses Orleans, Cosmos, and Kafka to consume, process, and publish real-time electrical consumption and situational awareness for our devices.

Ping me if you want more information

Rich

[ElderJames]

Hi Rich,

In my case I just need to use Orleans to host MQTT Broker and run at the edge. So I wanted to know how to use Orleans' clustering capabilities to implement clusters of other managed services (like hosted services).

[fuocor]

We actually isolated all non-orleans hosted services outside of orleans. Those services consume orleans functionality as would any client. The reason why we do not co-host them is to allow each service to scale independently.

[staviloglu]

Hi @fuocor.
I also started with separate services consuming Rabbit queues to scale independently from Orleans silos, but when we needed to preserve packet order this didn't workout. Scaling consumers would break packet order.
Does your project requires preserving packet order, if yes how did you manage to make sure that grains are processing messages in order?

[staviloglu]

I started thinking of a solution where a Silo has single RabbitMQ connection and each grain creates its own channel to consume its specific queue on Rabbit. I think that I can preserve packet order and scale by adding more silos to the cluster to support more devices as needed. Would like to know your thoughts about this.

[staviloglu]

@johnds1974 I'm curious about what you mean by connected grain instances? Are you trying the same architecture that I described?
Are you sure the memory is used by Orleans runtime or some other code that you might be using inside grains?

I don't know if you have tried Orleans Dashboard but it is capable of showing plenty of silo counters and grain throughput & latency.

I haven't done load tests yet, but I will asap.

[johnds1974]

Hi @staviloglu I figured out my problem eventually. So my architecture is similar to yours in that I have a grain instance representing each connected client, basic stuff there. It took me several attempts to discover the cause of the massive memory consumption, which was that firstly I was only using 1 Silo instance while doing my tests, and that each Grain instance of the connected client was looking up another Grain acting as a logger. This logger Grain was being called by each Grain instance by a common name.

So I assumed that there would be only 1 instance of the Logger grain in the Silo process, which luckily, there was only 1. BUT, the fact that each of my 5000 other Grain instances were using clusterClientInstance.GetGrain<>("LOGGER") for example, this was the problem. Each grain was instantiating it's own IClusterClient instance from which to get the logger grain, and this caused the massive memory hit.

When I changed the other grains to have a Singlton instance of IClusterClient (setup in Silo host DI) passed in via DI, so that all the Grains could lookup whichever other grain they needed (in my case they were all just looking for the Logger grain), then memory consumption literally vanished!

I have not tested spreading this test scenario across 2 Silo instances to see if that makes any other difference, but I pushed my test to 10000 simultaniously connected clients and my single Silo only hit around 200MB Ram vs previously 10Gb! Happy days!

[staviloglu]

@johnds1974 I also am curious about why you use clusterClient rather than this.GrainFactory.GetGrain inside your grain to call another grain. If logger grain is in the same silo this should be the way to call another grain I guess.

[johnds1974]

Thanks for the suggestion, I will certainly give GrainFactory a try, providing it does the same job/thing as the ClusterClient does. My only concern is that because of grain placement I don't know which Silo the logger grain will reside in, and does GrainFactory traverse Silo's when trying to find a grain of which I may only want 1 instance running in the entire cluster?

[jeremylcarter]

Hi @staviloglu I figured out my problem eventually. So my architecture is similar to yours in that I have a grain instance representing each connected client, basic stuff there. It took me several attempts to discover the cause of the massive memory consumption, which was that firstly I was only using 1 Silo instance while doing my tests, and that each Grain instance of the connected client was looking up another Grain acting as a logger. This logger Grain was being called by each Grain instance by a common name.

So I assumed that there would be only 1 instance of the Logger grain in the Silo process, which luckily, there was only 1. BUT, the fact that each of my 5000 other Grain instances were using clusterClientInstance.GetGrain<>("LOGGER") for example, this was the problem. Each grain was instantiating it's own IClusterClient instance from which to get the logger grain, and this caused the massive memory hit.

When I changed the other grains to have a Singlton instance of IClusterClient (setup in Silo host DI) passed in via DI, so that all the Grains could lookup whichever other grain they needed (in my case they were all just looking for the Logger grain), then memory consumption literally vanished!

I have not tested spreading this test scenario across 2 Silo instances to see if that makes any other difference, but I pushed my test to 10000 simultaniously connected clients and my single Silo only hit around 200MB Ram vs previously 10Gb! Happy days!

This maps to the memory usage of our Orleans IoT platform. Its great to know that if we need to triple our workload its just a simple as adding more silos to the cluster as opposed to insane amounts of rearchitecture/devops from other architectures.