Need to create JMX-style commandline tool for manually modifying Akka.Cluster membership

[Aaronontheweb]

So I had the realization this weekend that there's a massive piece missing to Akka.Cluster: http://doc.akka.io/docs/akka/snapshot/java/cluster-usage.html#cluster-command-line-java

if we don't have auto-down turned on inside Akka.Cluster (and it's risky to turn it on, for reasons I'll explain) then today there's no way to really "remove" a node from a cluster without writing code that does it inside your Akka.NET applications.

One of the things that we need to add is a JMX-style commandline tool for being able to manually down / join / remove nodes from the cluster. Here's the script that JVM Akka ships along for this purpose: https://github.com/akka/akka/blob/master/akka-kernel/src/main/dist/bin/akka-cluster

This isn't an alien idea at all - other systems that use DynamoDb-style clustering have tools like this and expect developers to use them when taking down nodes or removing nodes that have experienced permanent failure. Cassandra, Riak, et al work this way.

So we need to create this sort of experience for Akka.Cluster. The awkward part is - we don't have anything similar to JMX built into .NET.

Implementation Ideas:

1. Build a generic command-line kernel for Akka.NET applications on top of Akka.IO - that way we can send a little bit of information over a UDP / TCP socket from a ScriptCS / F# script that we can wrap underneath a batch / bash script like this.
2. Do idea 1, but only for Akka.Cluster.
3. Do something at a little bit of a larger scale so it might be able to turn into something JMX-esque eventually. That's something that's a bit beyond the scope of Akka.NET, but hey - for server-side applications in .NET that could be pretty cool.

Any other ideas?

[Aaronontheweb]

Ah, I just received an email from Viktor Klang about this - apparently the JMX endpoints and the commandline tool aren't used very much.

So one thing we should be encouraging Akka.NET developers to do instead if they use Akka.Cluster is run this code when a node needs to leave a cluster:

Cluster.Get(system).Leave(address)

That will gracefully remove a node, and that command can be executed anywhere inside the cluster - not just on the node that's leaving: http://doc.akka.io/docs/akka/2.3.4/scala/cluster-usage.html#Automatic_vs__Manual_Downing

We can also, programmatically, remove a dead node if we wish:

Cluster.Get(system).Down(address)

A node that's been marked as down can't rejoin a cluster. It's dead-dead.

[sean-gilliam]

If we build such a tool, how would security be handled? Obviously you don't want every Tom, Dick and Jane developer downing a node because trololol.

[Aaronontheweb]

That's a great question @sean-gilliam - in Java land they are able to set JMX permissions and even require things like authentication certificates, but we don't have the benefit of any of that infrastructure in .NET-land.

So what I would propose are a set of configuration options inside Akka.Cluster that for supporting this management interface that do one or all of the following:

1. Support TLS (more on that later)
2. Requires authentication credentials
3. Restricts the range of IPs that can connect (i.e. make it so the listener only accepts requests from localhost)

What do you think about those options?

[sean-gilliam]

Those sound good.

What about authorization though? Do we roll with role-based, claims-based, or do we just say if they're authenticated they can administer the cluster?

Security is a bitch to get right, but we also don't want to read a headline like "MASSIVE AKKA.NET SECURITY HOLE FOUND!!!!!11111oneoneone" :)