Overhaul test suite to be compatible with pytest #1100
Comments
|
The first big thing that jumps out at me is that pytest is real mad at all the GSSAPI related tests, partly due to the tricks they use to be CLI-configurable (the GSSAPI feature owners test against real Kerberos installs, IIRC.) That leads us to two things:
|
|
Also finding this |
|
Also, the current impl of |
|
All these nasty ugly globals in here 😭 |
|
Doing my best not to get sidetracked by overall modernization, but just enough to get pytest working. E.g. not going nuts with more old string interpolation fixes; not immediately replacing the current nasty logic around SFTP tests' temporary folders; etc. That stuff will remain 'old' for now. However a lot needs doing anyway, e.g. pytest fixtures (for things like the SFTP client setup, which I think does fall beyond the pale & should be taken care of now) don't seem to work with explicitly-unittest-subclased test suites, but then that forces my hand re: |
|
Also, to my chagrin, trying not to refactor all the obvious copypasta in here 🤦♂️ |
|
Note to self: running 'big-sftp' test module by itself on master takes ~17s, and ~18s under pytest, so I'm not actually sure the When I slap a |
|
Grump, either approach is insufficient, once I turn to the main SFTP suite – in part because lots of tests explicitly redefine the value of the Gonna have to dig a bit more here. Hooray. |
|
Having intermittent hangs that seem like they only pop up in the SFTP suite, though they are much easier to trigger if they run as part of the main overall suite. Love this stuff, really gets me excited and definitely not at all frustrating or sapping my will to live 👍 ✨ The SFTP suite does pass 100% now so I think I am just gonna go run each other module by itself (another nice benefit of moving to pytest; previously one had to add boilerplate to a given test file to run it alone) and try to get everything else to where it can pass. Then we can worry about hangs and shit. |
|
Also, skipped tests (mostly gssapi ones) seem to emit a big gnarly path that has nothing to do with the actual test being skipped. Swell. At least it's just ugly. |
|
Rest of test modules appear to pass without modification, so, that's nice. Means I can defer updating them away from unittest-style, apparently, too? pytest doesn't seem to care if your suite mixes and matches discovery styles. So this leaves the SFTP module hangs as the main outstanding issue. Seems like Of note - using |
|
Seems the hangs are more likely to occur on Python 3, not too surprising, it's always been a bigger culprit when it comes to race condition type stuff. Honestly probably useful here, means I can run under 3 locally and have a better chance at repro'ing (tho it comes up pretty often under 2 too) Seems like the issue happens about as often regardless of whether the 'server' side of the fixture is scoped to session or not. I guess that's a good thing. |
|
Using an error-counter task and running just one test ( |
|
Digging into things more now. Quick sanity check: why the hell was pytest-timeout showing SO many threads (like...at least a dozen or two) when I only expect a handful to be in flight during any given test? Is that a potential sign of the state that is gumming things up, or do I just not fully understand what's going on? When I run a single big-sftp test on its lonesome and emit When I run more tests (e.g. both SFTP suites), by the time we get to the test I decorated (which is maybe a dozen tests in) there are dozens of threads. (and if I Ctrl-C, only then do I see all the debug logging about hitting EOF and shutting down - for basically all those threads.) So sure, there's a shutdown problem, but still doesn't explain why any of these threads are affecting any of the others (if that's even what is going on.) More digging. |
|
Weirdly, when I re-enable pytest-timeout (I had it off for a bit to make sure it was not confusing things) it shows the main thread hanging out in a long return chain from a network read - in a spot that shouldn't have actually been blocking on anything (being outside any of the lock-obtaining bits.) Digging more into it to make sure my timeout isn't too low (it was in a test that does e.g. 1024 reads in a loop) cuz I don't entirely get why the main thread would stop there instead of in a spot where it's waiting on one of the other threads. (Hm - maybe it's because there are enough other threads active that the main thread's just getting choked out by the GIL? Would explain why the issue never comes up in a single-test setup...?) With a 30s timeout the issue is harder to repro now - I don't seem to have written it down above but I remember setting it to 30s at one point and finding the issue still popping up. Maybe that was w/ the full suite and not just the sftp ones? (If I'm right about this being "caused" by just too many active threads, that would make sense...) Yea...even running the full suite a few times in a row, a 30s timeout isn't triggering any fails. Dropping it back down to 10s almost immediately times out. Implies that there really is no hang going on, just lots-o-threads slowing things way down (and no doubt exacerbated by pytest-timeout when it is in play.) So then the question is, was this happening under raw unittest with the old suite? I needed to do more comparisons anyway to make sure this switcheroo didn't lose any tests or massively inflate runtime... |
|
Hm seems too early to make that call, eg Travis running my latest HEAD is definitely still getting full-on 10 minute (MINUTE, not second) long hangs and erroring out. And that's with no pytest-timeout in play at all (I backed it out of git because I realized that using its thread backend, as we apparently must do, is not really worth the cost.) Only difference ought to be A) Linux vs Mac and B) I still haven't committed the rollback of Definitely having issues repro'ing now, even under Python 3, locally. EDIT: yea, when I go back to scoping that fixture to 'session' level (locally), repro happens again basically every time (at 10s timeout, and also at 30s timeout, again multiple times.) Partial stack of one spot where the main thread was at a (30s) timeout cutoff: File "/Users/jforcier/Code/oss/paramiko/tests/loop.py", line 54, in send
self.__mate.__feed(data)
File "/Users/jforcier/Code/oss/paramiko/tests/loop.py", line 84, in __feed
self.__cv.notifyAll()
File "/usr/local/var/pyenv/versions/3.5.2/lib/python3.5/threading.py", line 362, in notify_all
self.notify(len(self._waiters))
File "/usr/local/var/pyenv/versions/3.5.2/lib/python3.5/threading.py", line 345, in notify
waiters_to_notify = _deque(_islice(all_waiters, n))
File "/Users/jforcier/Code/oss/paramiko/paramiko/channel.py", line 130, in __del__
self.close()
File "/Users/jforcier/Code/oss/paramiko/paramiko/channel.py", line 655, in close
self.transport._send_user_message(m)
File "/Users/jforcier/Code/oss/paramiko/paramiko/transport.py", line 1720, in _send_user_message
self.clear_to_send_lock.acquire()I've seen it hang here a few times now, too (though it's not always here.) |
|
Got up to 47 full test suite iterations under Python 30s timeout w/ pytest-timeout, no timeouts. Travis also ran that commit no problem, tho I am gonna hit its 'rerun' button a few times too. Real bizarre but I'll take it. Still gotta do that comparison to master tho. EDIT: also, on Travis, up to at least 10x full cycles (so, the whole matrix) w/o any timeouts. |
|
Old suite, under Python 3.5, locally, with coverage enabled (w/o coverage saves maybe 2s but that feels within stdev): New suite, 3.5, locally: So that's disappointing. For now I assume it's due to the creation of a new test server for every test; the old setup (as noted way above) was presumably keeping it around for the whole test run? Also the stats line implies pytest is actually running an additional 6 or so tests, which is mildly amusing (its discovery is definitely more thorough than the old setup's was, which was basically explicit opt-in.) |
|
OK, the extras seem to be the GSSAPI tests - right - makes sense, the old test.py wouldn't even load those up. So we're good I think, except for the tests now being a lot slower :( |
|
Double checked that today, enabling session-scoped SFTP stub server is still the trigger for hangs, so at least that's consistent...>_> Using pytest's builtin timing stuff to examine slowness. By default (no session-scoped server), it looks like this (I arbitrarily cut off after 1 page of my terminal): So, SFTP being slow is no big surprise (makes me think it might be nice to truly stub it out so it does not touch disk...but that can be later) but I note many of the ~1-1.5s bits are in the setup for SFTP tests and not the actual execution. Hoping the timeout plugin lets me actually just treat timeouts as failures (prior to now I always ran with Also curious how the suite looks when I filter out the SFTP tests entirely on both branches - how much of the slow is pytest and how much is in this SFTP crap? |
|
Grump, yea, the thread timeout backend does kill the entire process (as documented) and the default signal one still isn't working for me (the deadlocks are strong & ignore even manual Ctrl-C; have to Ctrl-Z and kill every time - so it's no surprise that other signal approaches also fail to interrupt.) And the timing stuff only prints at the end, which is frustrating, I'd prefer it to print as we go. Slapped debuggery into the right pytest hooks to print duration capture as-we-go and as I suspected, with session-scoped stub server, the ~1.25s setup time cost only occurs on the 1st SFTP test and the rest are negligible (0.0Ns). If we assume we mark SFTP tests as being integration/slow tests and only run them on CI - how does this look? (I'm just using
So yea, I'm gonna see about making it so default by-hand tests skip SFTP tests and call it a day. I do not think trying to revert to the |
|
Decided to do the right thing and use a custom pytest 'slow' marker (as a decorator) to mark the actually slow suites and/or tests. This is both SFTP suites, as a whole, because eh why not; plus anything that runs over 1s normally. Brings |
|
OK, this is Done Enough, #1101 will suffice for later. Time to write NEW TESTS. |
Been kinda unhappy with Paramiko's handrolled semi-unittest-based
test.pysince I came on board, and ideally I'd like to end up using my rspec-esque test kit, but since the latter has been rewritten under pytest, just getting our suite working with pytest would be a great step forwards. Plus all the other reasons to do so: pytest is the leading test runner these days, many folks are familiar with it, lots of plugins to use, better tox integration for people who use that (not me, yet), etc.I am working on this right now as a prerequisite for a bunch of test-driven thorny auth crap (#387) so the ticket is just my usual keeping track of stuff for posterity.
The text was updated successfully, but these errors were encountered: