Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Fix hanging SSH connections to Cisco equipment #198
I've been asked by my employer to build a configuration management workflow using Trigger. Our network has around 60 active remote sites, with mostly Cisco switches and routers, and Trigger has worked pretty well for us so far.
The only irritant is that the SSH connections sometimes hang indefinitely without connecting. For most bits of equipment it's a roughly 10% chance, but today I found a router which has about 95% repeatability. So after some state-of-the-art asynchronous debugging (aka. injecting a lot of stack traces and crying), I found out that it was a Conch problem: the Cisco SSH server needs to have sent the first message before the client bombards it with Kex information (as per http://twistedmatrix.com/pipermail/twisted-python/2009-September/020352.html), or else it hangs.
Seeing as it's been 5 years and neither Cisco or Twisted seem to have fixed this, I've made a patch to fix this in TriggerSSHTransport.
added a commit
this pull request
Oct 16, 2014
Oct 16, 2014
1 check passed
referenced this pull request
Oct 16, 2014
Short version: we've had a lot of success in managing our Linux environments with Salt, so we've extended Trigger's NetDevices tree to be more like Salt pillars/states, and wrote a Cisco config generator in Jinja that feeds off NetDevices.
Basically we have a few hundred network devices out in the field, quite a few of which were unknown to the current ops team until we began scanning sites with PRTG/nmap. Thanks I guess to the Cisco Way of administration, there was no configuration management; changes were deployed manually line-by-line. Whatever was running in the field was the definitive (and sometimes only) copy of the config. Most of those config rollouts are old, inconsistent and have plenty of residual cruft (e.g. ancient sources for DNS/NTP, ports changed and left with the wrong description, utility ports that were originally user-facing and still have VoIP subnet + QOS crap attached...).
We're slowly moving towards a model where NetDevices contains all of the device-specific config in a structured template-friendly format (e.g. interface definitions), and can be pushed through a template to produce a full Cisco config. For rolling it out, we have a workflow script which shows the proposed changes to the device's config to make it match the template output ("show archive config differences [HTTP URL of template output] system:running-config"), then after confirmation carries out those changes ("configure replace [HTTP URL of template output] list").
We have other plans for this (e.g. hooking into our ESB for device reporting/tracking), but for now this is pretty great. Thanks for releasing such a mind-bogglingly useful framework, I really hope it catches on.
@scottp-dpaw Sorry about the delayed response. I think I need to fix my email.
I would love to see what you've done with NetDevices to make it more like Salt's pillars/states. Is that something you can share?! And the Cisco generator fed from ND, too? Hello!
That model you're working towards is one that I have been trying to work on for a while. I am really happy to know that you're doing exactly that and using Trigger to do it. This is the kind of stuff I think would be really beneficial to the Trigger community.
I would love to talk about this in more depth, with as much as you're willing to share. Please consider joining us on IRC!
#trigger on irc.freenode.net