Sling Devops Experiments, Volume 5: Automatically Crankstarting Minions
This is the fifth of a series of experiments about how Apache Sling can be made more devops-friendly.
Continuing from the Git-driven crank file monitoring mechanism demonstrated in the previous vol4 experiment, we configure the Orchestrator to automatically crankstart new Sling instances for each new version of this crank file. This fully automates the update process.
The full test scenario is as follows:
- The Orchestrator is crankstarted and given the Git repository URL of the Minion crank file.
- The Orchestrator watches that file in Git for new versions. Once a new version
Vis found (or the initial version, when starting), the Orchestrator crankstarts
nSling instances from that version. This crank file has a few variables: Sling HTTP port number, MongoDB URL, ZooKeeper URL, etc. which the Orchestrator may optionally set.
- The Sling instances start and announce themselves to the Orchestrator when ready.
- Orchestrator has a target version
Vthat it wants to expose via the HTTP front-end. Once
nSling instances have announced themselves with that target version, the Orchestrator activates them atomically on the front-end.
- When old Sling instances are not needed anymore, they are killed.
The difference with the previous prototype is that the Orchestrator now automatically starts Minions, using Crankstart. This mechanism requires that the Orchestrator itself be crankstarted because that is how the Orchestrator is able to determine the path to the Crankstart Launcher JAR. When a set of Minions is no longer needed, the Orchestrator also automatically stops them.
The start/stop mechanism was implemented using the Apache Commons Exec library.
Before running this prototype, it is necessary to prepare your local Maven repository (using JDK 7 or higher):
mvn clean installthis project.
- Build the required snapshot bundles:
- Checkout Sling trunk at revision 1609716.
mvn clean installthe following paths:
- Navigate to the
sample-bundledirectory and build two versions of the sample bundle:
mvn -P1 clean install
mvn -P2 clean install
It is necessary to set up your environment for running the experiment. The following components are necessary:
Vagrant will create a virtual machine with the IP address 10.10.10.10, running Ubuntu with the above components already configured for the experiment.
Vagrantfile and all other necessary files are in the
Before launching the VM:
CRANKSTART_LAUNCHER_PATHvariable on line 121 of
Vagrantfileto point to the Crankstart Launcher JAR built above.
- If your local Maven repository is not under
~/.m2/repository, change the variable
MAVEN_REPO_PATHon line 122 accordingly.
Then, bring the VM up (from the
After Vagrant is done, you can
ssh to the VM:
When the VM is no longer necessary, you can shut it down:
or destroy it forever:
In Vagrant terms, the process of configuring a plain VM (installing packages, running scripts, etc.) is called provisioning. By default, provisioning is only done the first time you bring up a VM. In case you want to re-provision it (which would bring it to the same state it was when it was first created, with clean MongoDB, ZooKeeper, and the Git repository), you can use the
--provision flag when bringing it up or
if it is already running.
For running the experiment on this VM:
sudo su root
- Git repository for the Orchestrator is ready at
/home/vagrant/testrepoand already has an uncommitted
sling-minion.crank.txtfile in it.
sling-orch.crank.txtfile is also under
- Crankstart Launcher JAR is at
- Balancer configuration file for Apache Server is specified as
- HTTP front-end runs on the default port 80.
Launching the Orchestrator
Crankstart the Orchestrator:
java -Dgit_repo=<path-to-git-repo> -Dhttpd_balancer_config=<path-to-balancer-config-file> -jar <crankstart-launcher>.jar <sling-orch.crank.txt>
<path-to-git-repo>is the Git repository in which the Orchestrator will monitor
<path-to-balancer-config-fileis the path to the Balancer configuration file as specified in
<crankstart-launcher.jar>is the path to the Crankstart Launcher JAR
<sling-orch.crank.txt>is the path to it
i.e. on Vagrant:
java -Dgit_repo=/home/vagrant/testrepo -Dhttpd_balancer_config=/etc/apache2/mod_proxy_balancer.conf -jar crankstart-launcher.jar sling-orch.crank.txt
Additionally, you may specify additional parameters using the same
-D switches before the
httpd: path to the
httpd, i.e. assumed to be on your
port: port on which to start the Orchestrator (default: 1240)
n: number of Minions to start for each config (default: 2)
zk_conn_string: ZooKeeper connection string (default:
Once the Orchestrator loads, you should be able to access its status page at
/orch/status.html, i.e. http://10.10.10.10:1240/orch/status.html on Vagrant.
Executing the Scenario
sling-minion.crank.txt file to the Git repository. The Orchestrator should detect the commit and start
n Minions from it. You should be able to monitor the progress from the status page.
Once the Minions are ready and the Orchestrator activates the front-end, you should see something at
/mynode.test of your HTTP frontend, i.e. http://10.10.10.10/mynode.test on Vagrant. The
test script rendering the node displays three things:
- The content of the node
- The version of the
- The version of a test OSGi service
test script and the test OSGi service are supplied by the
org.apache.sling.samples.test bundle specified in the Minion crank file.
Now modify the version of this bundle from 0.0.1 to 0.0.2 and commit the crank file again. The Orchestrator should detect the change and start
n more Minions from the new file. Once these new Minions announce their readiness to the Orchestrator, the HTTP front-end should switch to them and the Orchestrator should terminate the old Minions. The updated bundle supplies updated versions of the
test script and the test OSGi service, and you should see respective changes at
The switch on the front-end is atomic (there is no point in time at which some of the elements rendered on the page come from the new Minion version and some from the old) and zero-downtime.
Note: Every time before re-running the experiment, it is necessary to delete the
sling-orch-crankstart folder created when running the Orchestrator. This folder is used as the Sling home directory of the Orchestrator, and restarting Sling from an existing home directory is not yet supported with Crankstart.
To verify that the switch between the Sling configs is atomic and consistent (from the client point of view), the
HttpResourceMonitor tool from the
tools directory can be used. This tool sends an HTTP request over and over in a single thread and logs changes in responses.
HttpResourceMonitor address [numThreads]
numThreads is the number of threads to send requests from and is 10 by default.
To monitor the output of the
test script on the Vagrant VM:
java -cp target/org.apache.sling.devops.tools-0.0.1-SNAPSHOT.jar org.apache.sling.devops.tools.HttpResourceMonitor http://10.10.10.10/mynode.test