Hanging when ssh'ing into a VM w/o pty=True

[GandaG]

Invoke version 0.12.2, Ubuntu Wily (15.10 I believe) 64 bits.

The method used:

@task
def enter():
    run("vagrant ssh -- -X -t 'cd /vagrant/; /bin/bash'")

The command run: inv enter

The command itself runs normally and does what is expected of it. When used through invoke it just hangs (no output). Other invoke'd commands work fine.

If you need more info just point me to where it is.

[GandaG]

Actually it just hangs whenever it ssh's into a VM. I'm guessing it can't be used to do that then?

[joystein]

I have a similar issue when setting up ssh tunnels:

@task
def setup_ssh_tunnels():
    jumphost = "jumphost"
    run("ssh -fN -L localhost:local_port1:remotehost:remote_port1 %s" % jumphost)
    run("ssh -fN -L localhost:local_port2:remotehost:remote_port2 %s" % jumphost)
    run("ssh -fN -L localhost:local_port3:remotehost:remote_port3 %s" % jumphost)

When running this task it will just hang on the first run command.

[GandaG]

Fixed by adding pty=True at the end of the run function.

@task
def enter():
    run("vagrant ssh -- -Xt 'cd /vagrant/; /bin/bash'", pty=True)

[bitprophet]

Thanks for reporting this - yes, forcing use of a pty often helps with otherwise bizarre issues, some applications misbehave without one.

I'm still curious why this is happening though, no obvious cause is springing to mind (most of them would usually involve either an error or an exit, not a hang). Going to reopen in hopes I have time to investigate sometime; it's possible it's highlighting a bug on our end.

[GandaG]

In this case, even with pty=True there were a few errors (the prompt was limited to 100~ characters, after that it glitched - the line turned invisible and deleted the previous line) which I fixed by replacing run() with os.system()

@task
def enter():
    import os
    os.system("vagrant ssh -- -Yt 'cd /vagrant/; /bin/bash'")

[bitprophet]

Sure, that's because run does a lot of stuff os.system does not & cannot do (print+capture, prompt response, etc). Unfortunately it means there are occasionally still frustrating quirks.

If you have time can you paste in some concrete examples of what your terminal looks like in these situations? E.g. what your $PS1 or similar is, etc.

For funsies I poked a bit on my end (I too have a nontrivial prompt setup - it's zsh and prints your usual path, username, git status, etc etc all one one line, then newline + chevron for actual 'prompt'). You can ignore a lot of this as it's mostly rambling to myself, but here's the notes:

• I can confirm the hang-without-pty, but I get an explicit, useful error message: Pseudo-terminal will not be allocated because stdin is not a terminal. - and an SSH session with no pseudo-terminal on the server isn't going to be very useful, I guess.
  • Curious why you didn't see this message on your end.
  • Note: My client & server are both my Mac (Yosemite) localhost. Haven't tried this on Linux yet.
• With pty=True, that message doesn't appear, but things still appear to hang.
  • ssh -v confirms it's connecting & spawning a shell
• Interestingly, with pty=False again, I tried typing in case there was still really a shell in play, and there is!
  • Things still behave funny in spots because there's no pty and thus all apps run in non-interactive mode: e.g. man doesn't page but simply prints the entire man page; ^D makes zsh spit an error, because zsh isn't in interactive mode; etc
  • But the shell is still definitely there and interpreting commands...with no prompt (again likely due to it running non-interactively).
• With pty=True, the above doesn't happen, just no activity until Ctrl-C.
• With INVOKE_DEBUG=1 and pty=True, I note that one of the IO threads is encountering a Unicode exception, which I bet is related to the unfinished encoding related work in #289 / #242 (my terminal is UTF-8 by default & my prompt contains Unicode)
  • EDIT: yea if I remove that Unicode char from my zshrc temporarily, I get a happy prompt (with pty=True) & all is well. Which brings me back to wondering which particular glitch @GandaG is having in their prompt :)

Going to leave this open as its own ticket for now but strongly suspect resolving the remaining issues in those tickets mentioned above, will help clear most of this up.

[redfacedquark]

Maybe this helps. I'm seeing both my shell get mangled (bash only, not on zsh) and my run commands (even "ls") fail to return (Ubuntu 14/15/? + Python3). I tried with and without context with no luck, with pty=False I get:

File "/home/agrajag/.virtualenvs/boat/lib/python3.6/site-packages/invoke/runners.py", line 813, in start
stdin=PIPE,
File "/usr/local/lib/python3.6/subprocess.py", line 957, in init
restore_signals, start_new_session)
File "/usr/local/lib/python3.6/subprocess.py", line 1484, in _execute_child
self.pid = _posixsubprocess.fork_exec(
AttributeError: 'NoneType' object has no attribute 'fork_exec'

My problem was that my tasks were along side my app and my app used gevent. Grabbing a constant from the app was causing gevent to monkey patch subrocess and _posixsubprocess.fork_exec didn't like that. My solution was to not import from my app in a way that caused gevent to monkey patch subprocess (in my case by moving the constant to utils, a non-gevent module that can be shared with tasks.py). Gevent being installed in the venv was not enough to trigger it, just monkey patching all.

Perhaps this issue could be addressed by detecting if gevent has been monkeying around and warn the user about potential issues. Happy to provide more feedback/debugging if you need it.

[bitprophet]

Back to this after doing #350. Interestingly, I can't even recreate this anymore on master, let alone the branch for #350. Possible some of the other work in the interim fixed it. (There was definitely a "make encoding errors explode always instead of hanging" fix I put in a few days ago on master, which is definitely related to this, but that doesn't explain why the actual encoding error we talked about above, didn't show up for me...)

But yea - I can do run("ssh -t myvm /bin/zsh", pty=True), get my dotfiles w/ funky prompt, and the shell works fine, no exceptions, no hang. True both on master and on my encoding branch. If you're happy grabbing from git and testing, be my guest, otherwise this work will be released as 0.13 in the near future. Closing this for now, please comment if you can still recreate hangs.

---

Note - pty=False still has the issues mentioned above, but that's a completely separate issue as far as I can tell, related to running a shell interactively while it is not in interactive mode (because there's no pty in play). There may never be anything we can do there besides make sure "use pty=True for interactive things!" is as loud as possible in the docs.