New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix hanging SSH connections to Cisco equipment #198

Merged
merged 1 commit into from Oct 16, 2014

Conversation

Projects
None yet
4 participants
@scottp-dpaw
Contributor

scottp-dpaw commented Sep 25, 2014

I've been asked by my employer to build a configuration management workflow using Trigger. Our network has around 60 active remote sites, with mostly Cisco switches and routers, and Trigger has worked pretty well for us so far.

The only irritant is that the SSH connections sometimes hang indefinitely without connecting. For most bits of equipment it's a roughly 10% chance, but today I found a router which has about 95% repeatability. So after some state-of-the-art asynchronous debugging (aka. injecting a lot of stack traces and crying), I found out that it was a Conch problem: the Cisco SSH server needs to have sent the first message before the client bombards it with Kex information (as per http://twistedmatrix.com/pipermail/twisted-python/2009-September/020352.html), or else it hangs.

Seeing as it's been 5 years and neither Cisco or Twisted seem to have fixed this, I've made a patch to fix this in TriggerSSHTransport.

@adonm

This comment has been minimized.

adonm commented Sep 26, 2014

+1

@jathanism

This comment has been minimized.

Member

jathanism commented Oct 9, 2014

Ohmigosh I can't believe I didn't even see this until now. I will give it a review and get back to you very soon! I'd love to talk more about your project, too!

jathanism added a commit that referenced this pull request Oct 16, 2014

Merge pull request #198 from scottp-dpaw/develop
Fix hanging SSH connections to Cisco equipment

@jathanism jathanism merged commit 389ad7f into trigger:develop Oct 16, 2014

1 check passed

continuous-integration/travis-ci The Travis CI build passed
Details
@jathanism

This comment has been minimized.

Member

jathanism commented Oct 16, 2014

I was hoping that this addressed a lame issue that is similar yet distinct w/ Force10 devices. (See #186)

Anyhow, thanks for the contribution!

@scottp-dpaw

This comment has been minimized.

Contributor

scottp-dpaw commented Oct 17, 2014

Thanks!

Short version: we've had a lot of success in managing our Linux environments with Salt, so we've extended Trigger's NetDevices tree to be more like Salt pillars/states, and wrote a Cisco config generator in Jinja that feeds off NetDevices.

Basically we have a few hundred network devices out in the field, quite a few of which were unknown to the current ops team until we began scanning sites with PRTG/nmap. Thanks I guess to the Cisco Way of administration, there was no configuration management; changes were deployed manually line-by-line. Whatever was running in the field was the definitive (and sometimes only) copy of the config. Most of those config rollouts are old, inconsistent and have plenty of residual cruft (e.g. ancient sources for DNS/NTP, ports changed and left with the wrong description, utility ports that were originally user-facing and still have VoIP subnet + QOS crap attached...).

We're slowly moving towards a model where NetDevices contains all of the device-specific config in a structured template-friendly format (e.g. interface definitions), and can be pushed through a template to produce a full Cisco config. For rolling it out, we have a workflow script which shows the proposed changes to the device's config to make it match the template output ("show archive config differences [HTTP URL of template output] system:running-config"), then after confirmation carries out those changes ("configure replace [HTTP URL of template output] list").

We have other plans for this (e.g. hooking into our ESB for device reporting/tracking), but for now this is pretty great. Thanks for releasing such a mind-bogglingly useful framework, I really hope it catches on.

@jathanism

This comment has been minimized.

Member

jathanism commented Jan 8, 2015

@scottp-dpaw Sorry about the delayed response. I think I need to fix my email.

I would love to see what you've done with NetDevices to make it more like Salt's pillars/states. Is that something you can share?! And the Cisco generator fed from ND, too? Hello!

That model you're working towards is one that I have been trying to work on for a while. I am really happy to know that you're doing exactly that and using Trigger to do it. This is the kind of stuff I think would be really beneficial to the Trigger community.

I would love to talk about this in more depth, with as much as you're willing to share. Please consider joining us on IRC!

#trigger on irc.freenode.net

@jamesraul

This comment has been minimized.

jamesraul commented Jan 9, 2015

James.raulinaitis@rakops.com
On Wed, Jan 7, 2015 at 9:23 PM Jathan McCollum notifications@github.com
wrote:

@scottp-dpaw https://github.com/scottp-dpaw Sorry about the delayed
response. I think I need to fix my email.

I would love to see what you've done with NetDevices to make it more like
Salt's pillars/states. Is that something you can share?! And the Cisco
generator fed from ND, too? Hello!

That model you're working towards is one that I have been trying to work
on for a while. I am really happy to know that you're doing exactly that
and using Trigger to do it. This is the kind of stuff I think would be
really beneficial to the Trigger community.

I would love to talk about this in more depth, with as much as you're
willing to share. Please consider joining us on IRC!

#trigger on irc.freenode.net


Reply to this email directly or view it on GitHub
#198 (comment).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment