Broadcast/per-silo grains

[ReubenBond]

In #587, we discussed the occasional desire to break location transparency of Virtual Actors and have the ability to message a grain (or otherwise) on an individual silo to broadcast a message to all silos.

In my case, I would have been happy to kick off a stream consumer inside a bootstrap provider and push messages to it using a stream, but this is not currently supported.

Use cases include:

• Configuration
• Cache management
• Metrics collection
• Fault injection
• Process control (eg, live code reload)
• Implementing certain distributed algorithms, eg Raft & Paxos

So, how should we do this?

[gabikliot]

@ReubenBond , is it only for "broadcast a message to all silos"? So from an API perspective you have a way to identify a group of grains and sending ONE msg to the whole group of grains and don't know how much are in the group and don't get to message each one individually?
This is a bit different from "break location transparency of Virtual Actors".

[ReubenBond]

@gabikliot for most of the use cases, yes. Although that would probably not be the nicest way to perform metrics collection. @jthelin likely has some ideas.

[gabikliot]

@ReubenBond , why not use the standard solution we recommend for this case:
Bootstrap provider makes a call into a PreferLcalPlacement grain with random id and that grain will subscribe to the stream?

[jthelin]

The other part of the "Per Silo" grain semantics is that those grain instances should never "roam" off their designated silo.

If that silo crashes, that grain is just gone, and would never be recreated on another silo.

A (simplified) concrete scenario example is if you have a "shard" of data physically located on one machine, then the grain responsible for dealing with that shard can only run on that one physical machine.

The current PreferLocalPLacement policy handles the initial "local" placement part, but not the "guaranteed no-roam" part.

I think my scenario is a slightly more specific and constrained version of the general requirement that @ReubenBond is describing above.

[gabikliot]

Do you guys have concrete ideas how the new API and semantics (w.r.t. failures) for this new type of grain might look like?

[jthelin]

From my perspective, the failure mode can be very simple:

If silo is gone, then the per-silo grain(s) in that silo are also gone.

I would ideally prefer a specific exception with well understood semantics (maybe something like GrainNoLongerActiveException ?) and [preferably] fast-track throw (as soon as the cluster membership change is detected) rather than needing to wait full timeout period for an eventual generic TimeoutException exception.
That makes it easier for any "coordinator" functions in my app to make better informed decisions about shard availability, rather than trying to second guessing whether timeout is from temporary communications glitch [retriable] or because silo just crashed / went offline [dead].

When new silo restarts on a failed host (i.e. same host / ip address but different silo generation number) then Orleans runtime will run the usual app boostrap provider plugins in that silo, which would recreate a fresh instance of the per-silo grain(s) on that silo.

[ReubenBond]

I am on-board with @jthelin's suggestion. It still requires us to add some application-specific logic in the bootstrap provider, though.

We could create a [SiloSingleton] marker attribute and automatically activate each marked grain type during bootstrap.

[jthelin]

BTW, the new [StatelessWorker(1)] functionality in PR #688 should now provide the "no-roam" guarantee i was talking about above, rather than using [PreferLocalPLacement] which is only best-efforts.

Updated: This approach does not work, for subtle reasons that @gabikliot pointed out to me in e-mail and hinted at below. :(

[veikkoeeva]

@jthelin An excellent point! Would it a worthy point to document in FAQ, for instance?

[jthelin]

I am hoping to upload a code sample this weekend, but I will look at the docs too ;)

[gabikliot]

Just to keep in mind, this is still StatelessWorker, which means you can have more than one globally. There will be up to 1 on every silo, but each silo can have 1 and globally there may be many.

Also, the above means you can not address an individual activation of the stateless worker globally. There is no way (currently, in the current implementation, unless we change the programming model) to send a message from silo X to StatelessWorker on silo Y. You can address local activation of the StatelessWorker. So for example, every BootstrapProvider will be talking to its own, single copy, of the StatelessWorker(1) grain.

[jthelin]

Yes, @gabikliot states a very important point to remember when using [StatelessWorker] attribute.

In my example scenario I use a different Guid (Grain ID) for each silo, so each of the PerSilo grains is independently registered / addressable.

I posted PR #738 with my minimal example code / scenario test case.

There are lots of similar ways to get PerSilo grains working, and lots of extra sophistication that could be added -- including higher-level programming abstractions like "DoPerSilo(Action)" -- but my example is intended to be a minimal usage example of the most basic PerSilo grain scenario.

Comments welcome :)

Updated: This approach does not work, for subtle reasons that @gabikliot pointed out to me in e-mail and hinted at above. :(

[jason-bragg]

Hi Jthelin,
For the purposes you described in #738, specifically

"Use cases include:
•Configuration
•Cache management
•Metrics collection
•Fault injection
•Process control (eg, live code reload)"

Have you considered the silo control channel Gabi added recently (#718)? It's a new capability wherein one can send commands to providers. You 'should' be able to create a provider that inherits from IControllable that processes commands that perform actions like you described. The glitch here is that these commands are executed on all silos, not a specific one. This can be worked around, but I'd not recommend that. Since it's a new capability, I thought I'd bring it up.

[gabikliot]

We can easily extend #718 to send a command to a provider on a specific silo. Silos list is exposed already via ManagementGrain, so adding such a capability would be trivial and it will not change any public grain abstractions, it's virtual nature or APIs.

[ReubenBond]

Interesting suggestion regarding #718 - I wonder how easily we could use this to implement a distributed consensus algorithm... could be a fun project for someone :)

EDIT: Upon inspection, the IControllable looks like it would be a pain to use for implementing anything non-trivial

[gabikliot]

@ReubenBond , why is that: "Upon inspection, the IControllable looks like it would be a pain to use for implementing anything non-trivial"?
I modeled it following @yevhen's "uniform interface" - just a weakly typed interface. One can build a strongly typed interface above it. It should not be more pain to implement anything non-trivial on top of IControllable as on top of F# or any non typed interface.

[ReubenBond]

Maybe we could have an IGrainWithSiloKey, what do you think?

[gabikliot]

And what happens when this silo dies?

[ReubenBond]

If the silo is no longer in the set of active silos, the call should fail with a well-typed exception

[gabikliot]

So non virtual actor, non location transparent? Plus, silo id is ip plus epoch, so exposing silo addresses should also be part of the programming model.

[ReubenBond]

The silo address itself is opaque. The epoch should probably be included if the epoch represents a new instance of a given silo.

I see the concern with breaking the abstraction. All abstractions are leaky, though, and this helps us to create cluster services which we otherwise will have a much harder time creating.

Is the main concern that IGrainWithSiloKey would make it too easy to enter into shark-infested waters?

[gabikliot]

We can definitely do that. In fact, we had this implemented about 3 years ago and removed afterwards. Totally doable to bring it back.

The question I have is can we do better.like you said, indeed, it pushes the developer into shark infested waters. It makes developers life much harder with those grains, in the scenarios they are needed.

Allowing more locality and broadcasting support in general is something I believe we need to do, as I commented multiple times in the past. The question is how.

I think we all, collectively, did not think hard enough how to solve this problem better. What you ate suggesting is easy to do, but I think we can do better. My intuition is that better solution exists. I might be wrong, and then we are back to what you suggested, but my intuition is we can do better. There is an idea of group grain for example, still fully virtual.

I would suggest to first explore the alternatives before we give up and go with what you suggested now.

Can you describe your exact scenario, and how you would solve it with your suggested non virtual actor, including error handling.

Btw, using steams from bootstrap provider (how this whole issue started) is something I already implemented. There is an open PR doing that.

[ReubenBond]

I agree with you, @gabikliot, we should look for the best solution possible.

Let's assume we want to implement a strongly consistent database atop Orleans. We intend to use a distributed consensus algorithm which involves persisting an operation log on each node.

When a node becomes available, we start replicating operations to it.
When a node is up-to-date, we allow it to take part in the replication quorum.
When a node fails or falls behind, we remove it from the quorum.
Clients can message an individual node in order to perform an operation (writes go to the leader, reads may be served from followers).

So in this case, I don't need to know the physical location of a node, I just need each node to be on a separate host/fault domain (if I form a quorum on a single host, I have not achieved fault tolerance).
I need nodes to be individually addressable so I can send messages to individuals ("hey, here's a new operation to replicate").
I need notification for when new nodes become available and when they become unavailable so that I can perform leader election and decide on the nodes included in the replica set.

Effectively, a reference to a node in this case is like a WeakReference<T>... a WeakGrainReference.

Tell me more about this "group grain"

[gabikliot]

I described the group grain above. You send one msg and it arrives to all group members. I didn't think about notifications about group changes, but it might be possible to add.

But I don't think it will work for your scenario. Since group grain I meant is an application abstraction. I think what you are describing is not an application layer, but rather a low level system layer and to built it you will need Orleans to expose low level, non application, data, like silos, pings, ...
Totally doable, maybe even a good idea, but not as application abstraction. I am of a strong opinion that if we start mixing application abstractions with lower level system abstraction we are doomed. Will become the next CORBA, WCF, SF: Too much options, too much confusion, too much layers intervene in each other .

To build what you are asking, we can just espouse system targets and our membership interface, with membership notifications. If we do that (not sure we should, but if) we just need to be very clear it is not an application abstraction: system target is not an actor. It's an endpoint. Different abstractions.

[ReubenBond]

I agree that we should avoid making these things too readily accessible - they are for building system services & not application-level use - unlike your group grain suggestion which satisfies most of the use cases in the initial comment.

Ideally we would expose the required functionality in a way which allows external plugins to take advantage of it while still dissuading application developers from going near it.

[sergeybykov]

One-per-silo application system targets, grain services, were implemented for #2444 in #2459/#2531.