Switch branches/tags
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
302 lines (277 sloc) 15.5 KB
<!DOCTYPE html>
<meta content="text/html; charset=UTF-8" http-equiv="content-type">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<meta name="viewport" content="initial-scale=1, maximum-scale=1">
<link href=",700,900%7CRoboto+Mono" rel="stylesheet">
<link rel="stylesheet" href="//">
<script src=""></script>
<script src=""></script>
<link rel="stylesheet" href="">
<script src=""></script>
<script src=""></script>
<script src=""></script>
<link rel="stylesheet" href="">
<script src=""></script>
<style type="text/css">
body {
font-family: 'Roboto', sans-serif;
code {
font-family: 'Roboto Mono', monospace;
header pre {
border:2px solid;
border-color: firebrick;
text-align: center;
header code {
ul {
list-style-type: square;
a {
a:hover {
.token.operator {
nav button {
box-shadow: 0 0 0 1px rgba(0,0,0,.15) inset,0 0 6px rgba(0,0,0,.2) inset;
font-family: inherit;
font-size: 100%;
padding: .5em 1em;
color: #444;
border: 1px solid #999;
border: 0 rgba(0,0,0,0);
background-color: #E6E6E6;
text-decoration: none;
border-radius: 2px;
nav.hide ul {
nav ul {
list-style-type: none;
nav li:before {
content: "- ";
figure {
text-align: center;
font-style: italic;
font-size: smaller;
text-indent: 0;
border: thin silver solid;
box-shadow: 1px 1px 3px black;
margin: 0.5em;
padding: 0.5em;
.clear { clear:both;}
<title>Feature Switches, Inheritance and Agile with Scala &amp; JMX on the JVM</title>
<meta name="twitter:title" content="Feature Switches, Inheritance and Agile with Scala &amp; JMX on the JVM">
<meta property="og:title" content="Feature Switches, Inheritance and Agile with Scala &amp; JMX on the JVM">
<meta property="og:url" content="/1609/feature-switches-agile-scala-jmx/">
<link rel="canonical" href="/1609/feature-switches-agile-scala-jmx/">
<!-- -->
<meta property="og:type" content="article">
<meta property="article:published_time" content="2016-09-30">
<meta property="article:modified_time" content="2017-07-29">
<meta property="og:description" content="William Narmontas takes you through Feature Switches, Inheritance and Agile with Scala &amp; JMX on the JVM.">
<meta itemprop="description" content="William Narmontas takes you through Feature Switches, Inheritance and Agile with Scala &amp; JMX on the JVM.">
<meta name="description" content="William Narmontas takes you through Feature Switches, Inheritance and Agile with Scala &amp; JMX on the JVM.">
<meta name="twitter:description" content="William Narmontas takes you through Feature Switches, Inheritance and Agile with Scala &amp; JMX on the JVM.">
<meta property="og:site_name" content="Scala William">
<link rel="author" href="">
<link rel="publisher" href="">
<meta itemprop="name" content="The most important Streaming abstraction">
<meta property="og:image" content="">
<meta itemprop="image" content="">
<meta name="twitter:image" content="">
<meta name="author" content="William Narmontas">
<meta name="twitter:card" content="summary">
<meta name="twitter:site" content="@ScalaWilliam">
<article class="h-entry">
<h1 class="p-name">Feature Switches, Inheritance and Agile with Scala &amp; JMX on the JVM</h1>
<h2>By <a href="/" class="u-author">William Narmontas</a>, <a href="/1609/feature-switches-agile-scala-jmx/" class="u-url"><time class="dt-published" datetime="2016-09-30T12:38:59.026Z">Sep 30, 2016</time></a></h2>
This page can be <a href="">edited on GitHub</a>.
<section class="e-content">
<p>Remember last time a critical service in production worked incorrectly and you
reverted your changes, or restarted the services for new configuration? Neither do I.</p>
<p>In one of my recent contracts, 10 days before production deployment I began to work on
a feature that would actually take 2&#8211;3 months. The management had not accounted for the complexity of the task, but no worries: we have Agile and
<a href="" target="_blank">Play</a> <a href="" target="_blank">Scala</a> at our disposal.</p>
<p>Once you know that you'll miss won't make a deadline, all risk is gone, and only
certainty is left.</p>
<p>I agreed with the project manager to implement the simplest, dumbest, safest thing
there was, so at least we would not miss the deadline. It would take 1 day to get into master.</p>
<p>This would be our basic interface:</p>
<pre class="language-scala"><code data-source="Feature.scala" data-from="1" data-to="3">trait Feature {
def execute(inputXml: Elem): Future[Result]
<p>And a very dumb implementation in Scala, called &#8216;StaticFeature&#8217;</p>
<pre class="language-scala"><code data-source="StaticFeature.scala" data-from="1" data-to="8">class StaticFeature extends Feature {
def execute(inputXml: Elem): Future[Result] = {
object StaticFeature {
val staticResult: Result = ???
<p>Now knowing that the management will be asking for constant updates for the final
deliverable, I agreed with the Product Owner that it'll make more sense to implement something simple that at least partially meets the
requirement, but would take a bit less. It took 2 weeks and we we were able to deploy it to production quite easily. If anything went wrong, we'd
be able to switch back to something we agreed to earlier (ie &#8216;StaticFeature&#8217;).</p>
<p>Welcome to &#8216;AdvancedFeature&#8217;, which is 10x more complicated than &#8216;StaticFeature&#8217;, is
tested, and can fall back to &#8216;StaticFeature&#8217; when it is unable to fit the bill.</p>
<pre class="language-scala"><code data-source="AdvancedFeature.scala" data-from="1" data-to="12">class AdvancedFeature(staticFeature: StaticFeature)
extends Feature {
def execute(inputXml: Elem): Future[Result] = {
.recoverWith {
case NonFatal(e) =&gt; staticFeature.execute(inputXml)
object AdvancedFeature {
def process(inputXml: Elem): Result = ???
<p>But let's say rather than fail and fall back, &#8216;AdvancedFeature&#8217; returned incorrect
output? Because the input XML is very loose, there was no guarantee that everything would work as expected and none of it could be easily unit
<p>However, how do we switch back? Revert Git code? Redeploy a configuration change? Come
on! We have the JVM at our disposal. It has something called JMX (&#8220;<a href="" target="_blank">Java Management Extensions</a>&#8221;) which
provides you with a way to manage your application at runtime. You can do metrics, rebind ports, change configuration options, change logging
verbosity, run diagnostics&#8202;&#8212;&#8202;all sorts of things. So here's how you do it:</p>
<p>First you write a generic interface, a &#8220;Management Bean&#8221;:</p>
<pre class="language-scala"><code data-source="FeatureConfiguratorMBean.scala" data-from="1" data-to="5">package features
trait FeatureConfiguratorMBean {
def getFeatureLevel: String
def setFeatureLevel(name: String): Unit
<p>Then you write an implementation which registers this component to the JMX
registry and allows its methods to be called remotely:</p>
<pre class="language-scala"><code data-source="FeatureConfigurator.scala" data-from="1" data-to="11">package features
class FeatureConfigurator() extends FeatureConfiguratorMBean {
val platformMBeanServer = ManagementFactory.getPlatformMBeanServer
val objectName = new ObjectName("app:type=FeatureConfigurator")
platformMBeanServer.registerMBean(this, objectName)
def stop(): Unit = {
def getFeatureLevel: String = ???
def setFeatureLevel(name: String): Unit = ???
<p>And now when your app runs, you open up <a href="" target="_blank">Java Mission Control</a> (jmc in UNIX shell), connect to the app, and then you can change values immediately (it'd
call setFeatureLevel method upon pressing Return):</p>
<!--*gXqhtsEU8DAl5QcDz9cdDQ.png -->
<img src="image1.png" width="700px">
<p>Of course in production you might want to have something easier to use, in which
case I used <a href="" target="_blank">Java 8's JavaScript interpreter jjs</a> to connect to a remote running process and change the value of configuration via a Shell
<p>Now the business had the certainty that instead of being stuck with incorrect behaviour
they'll get a lesser version which gives the expected behaviour.</p>
<p>Final implementation, &#8216;ComplexFeature&#8217;:</p>
<pre class="language-scala"><code data-source="ComplexFeature.scala" data-from="1" data-to="14">class ComplexFeature(serviceA: ServiceA, serviceB: ServiceB,
advancedFeature: AdvancedFeature)
extends Feature {
def execute(inputXml: Elem): Future[Result] = {
Future(ComplexFeature.process(serviceA, serviceB))
.recoverWith {
case NonFatal(e) =&gt; advancedFeature.execute(inputXml)
object ComplexFeature {
def process(serviceA: ServiceA, serviceB: ServiceB):
Future[Result] = ???
<p>This was the most complicated by far, and the business wanted to do an A-B type of
roll out of the feature into several production environments. Run on the first environment for a few weeks, then run it on the second, then the
third, and then we're done with the fourth. Should anything go wrong, we resort to the lesser versions and change the level at runtime. Easy &amp;
<p>This approach was successful. Ops loved it. PO loved it.</p>
<li>If feature C started producing incorrect results, it would fall back to B.</li>
<li>If feature B started producing incorrect results, it would fall back to A.</li>
<li>Ops were able to switch between these levels at runtime.</li>
<li>Ops were able to deploy the same code to different production environments and A/B
test the changes.</li>
<li>Ops and Test were able to deploy the same code to production, staging and test
environments and verify the behaviour of every single feature.</li>
<li>Ops were able to monitor behaviour and the rates of fall back on DataDog.</li>
<p><strong>Something now is almost always better
than nothing now.</strong></p>
<p>But everything now is better than something now, so choose Scala &amp; JDK 8, it has it
<p>Connect up with me <a href="" target="_blank">on LinkedIn</a> and <a href="" target="_blank">Twitter</a>.</p>
<p><strong>Update, <time class="dt-updated" datetime="2016-11-04T12:00:00">4 Nov 2016</time></strong>: A reader
gave a very good question:</p>
So what are the downsides and upsides compared to, say, database switches?
<p>My answer:</p>
No downsides compared to database switches, only upsides. The thing you want to work out is how to execute this trigger since your service is
1. Java Mission Control via SSH tunnelling. You'll need to specify extra Java properties to allow that.<br>
2. A custom script to make the call from the server itself. This is the approach I took in the article. Tooling takes a bit of time to build.<br>
3. Launch something like Hawtio on the server: <a href="" target="_blank"></a> This is a very nice simple approach but
you'll need to secure access control to it. Needs Jolokia: <a href="" target="_blank"></a><br>
4. Use <a href="" target="_blank"></a> with your own administrative panel via REST. Where you want to abstract the low level details.
Choose #1 if your developers have access to production machines and know what they're doing (small team, few apps).<br>
Choose #2 if you DevOps and Dev are in close communication but Dev don't have access to production.<br>
Choose #3 if you have more applications running on the machine, maybe even several instances of the same app.<br>
Choose #4 if you are delivering to a customer who has their own Ops team and need special reliable control of your stuff.<br>
I think this will warrant a further article.<br>
Databases shouldn't really be used for configuration or management. But of course not all platforms support management facilities.
<div id="disqus_thread"></div>
var disqus_config = function () { = PAGE_URL; // Replace PAGE_URL with your page's canonical URL variable = PAGE_IDENTIFIER; // Replace PAGE_IDENTIFIER with your page's unique identifier variable
(function() { // DON'T EDIT BELOW THIS LINE
var d = document, s = d.createElement('script');
s.src = '';
s.setAttribute('data-timestamp', +new Date());
(d.head || d.body).appendChild(s);
<noscript>Please enable JavaScript to view the <a href="">comments powered by Disqus.</a></noscript>