Scaling SPKR

Initially, by default, SPKR runs on a very small cluster consisting of four shards hosted on a single server node. In order to add more nodes and scale horizontally, a few changes are required. This procedure will highlight the important steps in building a distributed cluster. At the end of this procedure, SPKR should be properly configured to run on multiple nodes.

Requirements

Your SPKR store must be completely empty. If you’ve used SPKR before, your database is not empty. A simple way to clean the database is to drop all tables from the schema. They will automatically be created at next launch. Scaling the cluster does not necessarily require you to flush all data, it can also be done through careful data replication (see slave bootstrap), combined with ZooKeeper configuration altering. In any case, these scaling operations should be well planned ahead of time in a real-world application.

Resulting cluster

Although this article does not impose any particular cluster structure, this is what the resulting cluster would look like given the examples.

1) ZooKeeper

“ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.” In the context of SPKR, it’s an external dependency used by NRV to dynamically manage the nodes in the cluster. It keeps track of the node associated to every service member for all services, including the SCN service, the MRY store service and the http api. One of the main features of zookeeper is its ability to scale on multiple nodes. However, for now, a single node will suffice. This node may either be local or remote. The first step is to download ZooKeeper, install it, and to run it. Once installed, make sure the ZooKeeper service is running. service zookeeper-server status will check the status.

2) Enabling the dynamic cluster

Before building the cluster, we must first disable the static configuration.

1. Open the SPKR configuration file “spkr.properties” located in “spkr/etc/”.

2. Disable the static cluster: “spkr.cluster.static = false”. The static cluster is custom cluster configuration created only to allow quick setup of the application and/or testing of other independent features. Unlike the dynamic zookeeper cluster, it does not make use of the core distributed features offered by NRV.

3. Depending on what kind of cluster you wish to build, you may have to change the “spkr.nrv.listen.port” and the “spkr.api.listen.port”. Basically, each node must have a unique ip/port combinaison. If you’re running multiple nodes on the same server (for testing purposes), you should use a different configuration file for each node, allowing you to specify unique ports for a same ip. This may required implementing configuration file specification through parameters.

4. Change “spkr.zookeeper.servers” to reflect the location of your ZooKeeper server(s). The default configuration assumes that your server is on the local machine. You’ll want to specify a valid hostname if you’re planning on using either multiple servers for your nodes or a remote ZK server.

3) Building the cluster

Zookeeper should now be running, but is completely empty. We’ll create a small NRV cluster configuration and push it to zookeeper. For this purpose, the file “spkr.cluster” located in “spkr/etc/” offers a cluster template that can be easily modified, and pushed to ZooKeeper using the NRV’s zookeeper tool (as explained below).

1. Open the “spkr.cluster” file located in “spkr/etc/”

2. Notice how the default configuration for SCN points to the local SPKR server. This default value turns SPKR’s SCN service into a service member capable of generating ids and timestamp. To properly scale our cluster, we need to extract the SCN service responsibility and run it on a dedicated node. The SCN framework comes with an stand alone SCN executable server. To run it, simply compile it “sbt clean compile stage” and run it using “./scn-core/target/start” using the following 2 vm arguments:

• Dscn.config=<path to your spkr’s spkr.properties config file in spkr/etc/>
• Dlog4j.configuration=<path to scn’s log4j.properties file (in “scn/etc/”)>

note: scn’s main class is ScnServer.scala

With and SCN service is running on a remote node, we can promote it to a master service member by changing the /services/scn/members/0=0:localhost:nrv=8888 in “spkr.cluster” to the appropriate node pointing. Note that by default, the SCN stand-alone service application uses the port 9595 defined in its configuration file (scn/etc/default.properties).

3. In the last section of the file, “SPKR MRY”, shards can be resized, moved to other nodes, merged, split, etc. Note that these things cannot be done at runtime (at least not without some restrictions), as data in the store would not be present on the new service member. Here’s an example of a simple cluster you could build to start with:

/services/mry.spkr=mry.spkr
/services/mry.spkr/members/2147483648=2147483648:<SERVER1_IP>:nrv=8888
/services/mry.spkr/members/2147483648/votes
/services/mry.spkr/members/4294967296=4294967296:<SERVER2_IP>:nrv=8888
/services/mry.spkr/members/4294967296/votes

In this example, two equally sized shards on two nodes are distributed on two distinct servers.

4) Applying the cluster to ZooKeeper

To apply the cluster configuration to ZooKeeper, we can use the NRV Zookeeper tool. This tool comes as an executable sub-project of the NRV framework called “nrv-zookeeper”. The tool can be used to interact with any zookeeper server. You can execute it using the “--help” parameter to learn more. To push your cluster configuration, your command should look like this:

./nrv-zookeeper/target/start -u <comma separated zookeeper servers hostnames> <spkr.cluster file path>

e.g.

./nrv-zookeeper/target/start -u 127.0.0.1/local ~/spkr/etc/spkr.cluster

note: to create the start script, use “sbt clean compile stage” in the nrv directory

This tool can also be used to:

• Display the content of ZooKeeper
• Compare the the content of ZooKeeper with your local version of the cluster file.
• Move a service member from a master replica to a slave replica (e.g. for manual failover) at runtime*.
• Split a shard into 2 halves at runtime, allowing the cluster to scale**.

*The target node should be a slave with an exact copy of the master’s store. **Should be done very carefully. Development is planned to make shard splitting at runtime easier.

5) Starting SPKR

You can now start SPKR. Both nodes need to be started (in no particular order). Both, the load and the data will be spread evenly between the nodes. Ideally, you should have a load balancer to redirect the traffic on either a random node or directly the right node. The NLB project (Nrv Load Balancer) is currently in development specifically for this purpose. Though this is a fairly easy challenge to solve, the SPKR javascript client queries systematically a single SPKR server, limiting SPKR’s ability to scale in its current form.

Once SPKR is up and running, you can use a tool such as VisualVM with the MBeans plugin to browse metrics on each node, and see how the load is spread. You can also look at the log4j output to see how the cluster reacts to the multiple nodes. And finally, you can check out the MySQL database content of some of the main tables, and observe, how the data is stored, depending on the shard it is associated to. See the SPKR database schema to better understand the sharding.