Get better SSH testing infrastructure

[bitprophet]

Description

Spinoff from #183, though that's mostly not important.

Need to test real, honest to God SSH level stuff, i.e. run() and its constituent underpinnings, if we're to have any chance of a serious test suite.

There are, as I see it, 3 ways to do this:

1. Create our own high-ish level mock of the Paramiko Channel object, since that's what our code interacts with, so that we can say "input X results in output Y".
   • Pros: it's a public API that won't change much
   • Cons: practically reimplementing the entire class structure of Paramiko, insofar as threading behavior and state-keeping goes. So far, this has not borne out anything fruitful (I tried.)
2. Mock/monkeypatch lower-level Paramiko objects, which right now looks like it'd be either Transport or Packetizer.
   • Pros: lowest possible level of monkeypatching means smallest potential amount of mock code, closest to "real" execution without actually requiring a real SSH server.
   • Cons: heavily tied to private or semiprivate APIs within Paramiko; possibly just as complicated to effect as the higher level mock, depending.
3. Use real SSH server.
   • Pros: best testing possible, as it's the real deal. No mocking needed, either.
   • Cons: tons:
     • Slows down tests, possibly by a lot if testing system is not adjacent network-wise to target system or test commands involve disk I/O.
     • Requires knowledge of state on target system in tests: return value of commands, sudo passwords, etc.
     • State might also need to be reset each time, which is subject to error.
     • Requires access to target system (potential security risk).
     • If target system is intended to be local VM and not a single specific machine, then the barrier to running tests becomes much higher than I'd like it to be. (Even if target system is simply localhost, then the user has to have a running sshd, which is common but not ubiquitous.)
     • If target is a distinct system, then tests will fail if system goes down or is unavailable or user is not online, so no more hacking on the airplane or out in the sticks.
     • Target could be CI server's localhost, which is relatively neat and all, but then we're leaning heavily on CI for full testing and that brings us back to "contributors cannot easily run the test suite".
     • Probably more I haven't even thought of yet.
4. Use fake SSH server. I.e. use Paramiko's server side functionality.
   • Pros: while we'd probably still need to actually run the server in another thread or process, it's still effectively mocking instead of hitting up a real system's sshd, as we'd be able to mock out what it does when it receives commands (I assume).
   • Cons: might not actually be possible to fake the commands, could be a decent amount of work depending on the API required, requires allocation of a network port (not a huge deal tho).

As indicated, I've taken a moderately sized stab at option 1 already and failed, and am currently investigating option 2. There's lots of support in IRC for option 3, including offers for a target testbed, which is good, though note all the cons. I'd very much rather stick with an in-code solution if it's not too painful. Option 4 just occurred to me now, though I recall coming up with it in the past at some point. May be worth looking at 4 before 3.

---

Originally submitted by Jeff Forcier (bitprophet) on 2010-06-22 at 04:29pm EDT

Relations

• Related to Consider forking Paramiko #275: Consider forking Paramiko
• Related to Prompts appear to kill capturing (now with bonus test server!) #183: Prompts appear to kill capturing (now with bonus test server!)

---

Closed as Done on 2010-06-23 at 05:35pm EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

Yea, option 4 -- use Paramiko's server mechanisms -- seems like the most sensible way to go, all things considered.

Currently hacking up the demo server a bunch and should soon have it boiled down to something compact-ish that can serve the needed purpose (basically, a command => stdout/stderr/status values mapping).

Main question in my mind right now is whether it's sufficient to spin up a new server instance for each test or if I should try and set up a long-lived server that can reopen new channels on demand. The demo server is a strictly one-shot deal so getting it to be longer-lived would require some tweaking. Hopefully not much though. OTOH, especially as we only have a handful of SSH related tests right now, it may be premature optimization.

---

on 2010-06-23 at 12:19pm EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

Got base case working, hooray! Currently just doing the one-shot per function invocation, and needs one to wrap calls with disable_known_hosts=True, but it does seem to work.

---

on 2010-06-23 at 02:21pm EDT

[bitprophet]

Jeff Forcier (bitprophet) posted:

---

Test server is now sufficiently robust that I can close this and continue tweaking it as necessary as the test suite grows.

---

on 2010-06-23 at 05:35pm EDT