Local terminal stdin detatched if ThreadingGroup run includes sleep

[jensenak]

I'm using a threading group to run shell commands. After running a script that includes a sleep, the local terminal is left with stdin detached (keystrokes not visible on command line) and the terminal has to be reset.

I've tried this quite a few times and find that it only happens with ThreadingGroups (SerialGroups are okay). The sleep command can be anywhere in a one-liner (first command, middle, last) and can be joined in the one liner with either semicolons, or double ampersand. All of the commands run as expected, but the terminal remains in a bad state.

Strangely, if the previous run exited with an uncaught exception, the terminal will not be affected.

To reproduce:

from fabric import ThreadingGroup as Group

# raise ValueError()
remotes = Group("host1.example.com", "host2.example.com")
result = remotes.run("echo 1; sleep 1; echo 2")

Run the script above. After it exits, type something on the command line. If you see no output, <ctrl>+c and type reset<enter>. To see the post-exception behavior, uncomment the raise line, run the code, comment the line, and run twice more. The first successful run will leave the terminal in a good state. The second will leave stdin detached.

I discovered this issue with sleep in my testing, but it's possible other commands may have the same effect. There's also a chance I'm just doing something wrong. If that's the case, my apologies.

My setup:
python 3.6.4
fabric 2.1.3
OSX 10.13.5, connecting to Ubuntu 14.04

[bitprophet]

See #1814 as possible second reproducible problem case.

This sounds like a legit bug to me and I'm not sure what's causing it offhand. Smells like it could be a generic Unix issue with terminal pipes being attached to multiple subprocesses at once, or (esp going by #1814's example) a race condition around pipe state, or something like that.

Will try to reproduce and muddle out a cause/solution.

[bitprophet]

Also, this probably requires fixing at the Invoke level and may be purely in its domain (insofar as I simply haven't done much with threading in a pure-Invoke context yet; but see eg pyinvoke/invoke#194 - it's a thing that should happen there too). In that case I'll move this to a ticket there & the Fabric "fix" would be to upgrade one's Invoke once fix released.

[nicktimko]

I was on Ubuntu 16.04.2 connecting to the same.

[bitprophet]

Another report of same issue in #1829. This is in my next-bugfix milestone and I'll be focusing on that hopefully next OSS day (Mon).

[bitprophet]

I just tried to reproduce this (2.0 branch, Python 3.6.4, macOS 10.12) and was unable to, unfortunately. First tried double-localhost, then two separate remote cloud instances, no dice either way; my terminal's fine afterwards.

Going to try on a Linux container in a bit just in case that helps, but since the OP was also on macOS, not sure it'll make a difference. I'll also try running it in a loop to see if it's only an occasional repro.

I'll also try it on 2.1 in case we somehow introduced it in 2.1, tho this seems very unlikely.

@jensenak @nicktimko are you reproducing this 100% of the time? 50%? 5%?

[nicktimko]

@bitprophet on 2.1.3 it was happening in my actual workflow fairly often (>80%, I was also going in parallel to 6 servers, not 2), though in my contrived example from #1814 it's much lower, maybe 20%. I can try to come up with a Docker setup, or failing that a Vagrant setup to repro.

[jensenak]

@bitprophet This has been 100% of the time for me. Just to be sure, I started a fresh virtualenv with only fabric installed. I tested 2.0, 2.1, and 2.2. The example code I pasted produced the described behavior each time. In all tests, I was connecting to Ubuntu 14.04 remotes.

I am on a different version of OSX (10.13). Perhaps that's related? Though @nicktimko was not on OSX at all.

In case another version is an issue, here's what pip freeze looked like in my virtualenv:

asn1crypto==0.24.0
bcrypt==3.1.4
cffi==1.11.5
cryptography==2.3
fabric==2.2.1
idna==2.7
invoke==1.1.0
paramiko==2.4.1
pyasn1==0.4.4
pycparser==2.18
PyNaCl==1.2.1
six==1.11.0

Seeing as all these were installed as dependencies of fabric 2.2, I'd expect your versions will look similar.

If there's more I can do to help, I'm more than willing. Just not really sure where else to look.

[nicktimko]

What commit should I test with; have you made any changes recently that might affect things? I'll try with the freeze above, you could also provide another frozen reqs.txt and I can see if that works/doesn't for me.

[bitprophet]

@nicktimko @jensenak Thanks for the extra info. I'll keep trying to repro it here; at 20% I'd definitely not have tried it enough to trigger. My remotes have been Mac and some older Debians, I can try Ubuntu Trusty in case it's somehow specific to that (which would be odd, but hey, this whole thing is odd.)

Also, what's your local shell environments? Mine is zsh on the (again, macOS 10.12) builtin Terminal.app, inside tmux. I'll also try some permutations around that angle in a bit.

[bitprophet]

AHA. This seems to be bash specific! Was still unable to repro under zsh outside of tmux, but the moment I try under bash, I immediately get the symptoms mentioned. Ditto inside of tmux, so tmux has no bearing - it's a shell thing.

Why this would behave differently under bash vs zsh, I have no immediate idea. Could be specific to how they are implemented, or (seems more likely) maybe something in my zsh dotfiles is preventing the problem? Will have to dig...though identifying a solution on the Python side is necessary either way most likely.

EDIT: also, the reproduction happens even when connecting to my localhost's sshd multiple times concurrently, which isn't too surprising. So the remote end seems to not matter.