automation-gone-wild.html

---
date: 2016-08-25T09:02:00.000+02:00
tags:
- automation
- workshop
title: Automation Gone Wild
url: /2016/08/automation-gone-wild/
---

<p>My “<em><a href="/2016/08/and-this-is-why-you-need-automation/">this is why you need automation</a>” </em>blog post triggered numerous comments and tweets. I loved this one:</p>
<blockquote class="cite">What if the mistake was embedded into the automation process/tool (designed by humans) in the first place? Now you have a video series titled "Automation Gone Wild".</blockquote>
<p>I guess this tweet is a priceless answer to that question:<!--more--></p>
<blockquote class="twitter-tweet" data-lang="en"><p dir="ltr" lang="en">To make error is human. To propagate error to all server in automatic way is <a href="https://twitter.com/hashtag/devops?src=hash">#devops</a>.</p>
— DevOps Borat (@DEVOPS_BORAT) <a href="https://twitter.com/DEVOPS_BORAT/status/41587168870797312">February 26, 2011</a></blockquote>
<script async="" charset="utf-8" src="//platform.twitter.com/widgets.js"></script><p>On a more serious note, I’ve heard plenty of horror stories when delivering my <a href="http://www.ipspace.net/Hands-On_Network_Automation">Network Automation</a> and <a href="http://www.ipspace.net/SDN,_OpenFlow_and_NFV_Workshop">SDN workshops</a>. Here are just two of them.</p>
<h4>Automating Cisco IOS Software Updates</h4><p>A large financial organization automated IOS software updates – routers across hundreds of branch offices were regularly upgraded to latest software release overnight.</p>
<p>Everything worked great until someone made a typo and specified an .md5 file instead of a .bin file in the upgrade request, bricking hundreds of remote routers.</p>
<p>The next few days were spent frantically running around and downloading the software to the routers through the console port.</p>
<p><strong>Lesson Learned</strong>: check the inputs. Make changes gradually (Tom Hollingsworth recommends <a href="https://networkingnerd.net/2016/04/20/automating-change-with-help-from-fibonacci/">Fibonacci sequences</a>).</p>
<h4>Automating Firewall Policy Deployment</h4><p>A retail organization used vendor firewall management tool to manage firewalls on hundreds of small sites around the country. They also used that tool to deploy changes to firewall and VPN policies.</p>
<p>The network administrator assumed that the software might fail, so he split the sites into a dozen of groups, and spread sites in each group around the country, so that a failure of one group wouldn’t bring a whole region down.</p>
<p>Not surprisingly, a failure did happen due to some weird glitch in the management software, and updating one of the groups failed, leaving those firewalls in some weird inaccessible state.</p>
<p>Now the resilience built into the whole system worked against the poor network admin – he had to travel around the whole country fixing firewalls belonging to that particular group. Fortunately, the country in question wasn’t US or China ;)</p>
<p><strong>Lesson Learned</strong>: things will fail. Always have a Plan B which does not involve driving around. <strong>reload in 5</strong> is your friend.</p>
<h4>Now what?</h4><p>Based on horror stories like this, you might conclude that you don’t want to touch the network automation with a 10-foot pole. I’m positive the bank tellers had the same reaction when banks introduced the first computers; how could you possibly trust that thing to balance the books properly.</p>
<p>In the meantime, we made huge progress in software development processes, and if you want to have a better chance of a working solution you should use those processes when developing your network automation solutions. </p>
<p class="note">Just because we know how to develop software properly doesn’t mean that your organization is doing it, or that people aren’t cutting corners. Then there are the <a href="http://www.daedtech.com/how-developers-stop-learning-rise-of-the-expert-beginner/">expert beginners</a> and <a href="http://thedailywtf.com/">software development horror stories</a>.</p>
<p><strong>Summary</strong>: You MUST treat the network automation like any other mission-critical application and use the same processes in its development (source code/version control, unit tests, integration tests...); implementing it with a series of scripts hacked together by a programmer wannabe (aka networking engineer) over a weekend will get you the results you deserve.</p>
<p>Also, if you want to know more about network automation, check out my <a href="http://www.ipspace.net/Hands-On_Network_Automation">workshop</a> and <a href="http://www.ipspace.net/Roadmap/Network_Automation_webinars">webinars</a>.</p>