Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pxssh: sendline()ing this string results in weird outputs #137

Closed
erikbgithub opened this issue Nov 19, 2014 · 6 comments
Closed

pxssh: sendline()ing this string results in weird outputs #137

erikbgithub opened this issue Nov 19, 2014 · 6 comments

Comments

@erikbgithub
Copy link
Contributor

Hi guys,

I've found a very weird situation, which might be a bug or not, not sure.

import pexpect.pxssh as pxssh
# do the login
ssh.sendline('ls -al /etc/hipos/hipos-device.recorder; echo "<retcode>$?</retcode>"')
ssh.prompt()
print ssh.before.encode('string-escape')

results in the following awesome string on my up to date Ubuntu machine:

ls -al /etc/hipos/hipos-device.recorder; echo "<retcode>$?</retcode>" \r\x1b[A\x1b[C\x1b[C\x1b[C
\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C
\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C
\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C
\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C
\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C
\x1b[C\x1b[K"\r\nls: cannot access /etc/hipos/hipos-device.recorder: No such file or directory
\r\n<retcode>2</retcode>\r\n

(all in one line, broken down for better readability)

On the embedded systems we are using pexpect to interact with, the following output gets printed (there the file sometimes exists):

ls -al /etc/hipos/hipos-device.recorder; echo "<retcode>$?</retcode>" \r\x1b[A

And after that output it's not possible to reuse the same pxssh object.

If I change anything in the input, e.g. remove the echo of the returncode, or the a from the ls -al everything works fine and I don't get any unnecesary control characters.

Any idea how I could investigate about this problem?

@jquast
Copy link
Member

jquast commented Nov 19, 2014

Clarify: You don't want to see any terminal sequences at all, is that it?

(You state another problem, "And after that output it's not possible to reuse the same pxssh object." and I'm not sure I understand. You would have to demonstrate the error, or maybe its a timeout, or perhaps it fails to synchronize the prompt... )

Firstly, You may want to add keyword argument echo=False in your pxssh.spawn() call to prevent seeing your commands in output twice.

You're always going to get terminal control sequences, because that's the purpose of pexpect: to provide a real terminal in such cases it is needed. All remote commands executed (such as ls(1)) will identify you as a real terminal and expect that you support various operations and control sequences. I'm afraid there is not any way to turn it off in pexpect -- it is the basis of its purpose.

  • It might be that pexpect is not the solution you are seeking -- you might better seek out paramiko or fabric, which provides execution of remote commands without a tty.

Unless of course you need a tty, like for interactively executing something that expects a terminal -- it is just the sequences that bother you. Well you're on the right tack with pyte, but:

  • pexpect does offer facilities for simple sequence interpretation:
>>> from pexpect.ANSI import ANSI
>>> ans = ANSI(r=24, c=80)
>>> ans.write('ls -al /etc/hipos/hipos-device.recorder; echo "<retcode>$?</retcode>" \r\x1b[A\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[K"\r\nls: cannot access /etc/hipos/hipos-device.recorder: No such file or directory \r\n<retcode>2</retcode>\r\n')
>>> print repr(ans.dump().strip())
u'ls -al /etc/hipos/hipos-device.recorder; echo "<retcode>$?</retcode>"         "                                                                                ls: cannot access /etc/hipos/hipos-device.recorder: No such file or directory   <retcode>2</retcode>'
  • I have another library 'blessed', where this could also solve your issue:
>>> from blessed import Terminal
>>> repr(Terminal(kind='xterm-256color').strip('ls -al /etc/hipos/hipos-device.recorder; echo "<retcode>$?</retcode>" \r\x1b[A\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[C\x1b[K"\r\nls: cannot access /etc/hipos/hipos-device.recorder: No such file or directory \r\n<retcode>2</retcode>\r\n'))
u'ls -al /etc/hipos/hipos-device.recorder; echo "<retcode>$?</retcode>"         "                                                                                ls: cannot access /etc/hipos/hipos-device.recorder: No such file or directory   <retcode>2</retcode>'

and pyte too, which seems like maybe the best there is available for interpreting the full xterm feature set.

If you're curious, the terminal sequences you see are basically: \x1b[A (up, one time), \x1b[C (right, 79 times), \x1b[K (move to column 0). It's some sort of optimization that the shell provides before executing commands, I think it might be a workaround about whether it is a multi-line prompt, or whether the user's control character was echoed literally. I would guess you're not using bash, I've never seen bash do that before. I also recognize that "$?", its for a remote cmd execution protocol very different from ssh, like interwoven teamsite related? God bless you and good luck !

@erikbgithub
Copy link
Contributor Author

First of all, thanks for your time. I'm so glad to speak to someone who knows something about terminals.

Well, I didn't choose pexpect for its Terminal I/O but for its sequencing of communication (send->expect->send->expect) which was something that I manually implemented with pyserial and pyssh in a previous version of my code (which was rather nasty for other reasons). Why pexpect needs to talk to me like I'm a terminal is something I never understood. I don't need it, but the API of pexpect simply is better than what I've seen in other libraries.

About your provided solutions, they all consume all of the command characters that's already quite good. Is there a way to tell a terminal to recognize them, consume them, but not do what they are telling the terminal to do, i.e., not to move up once, not move to the right and not move to column 0?

The echo $? echoes the returncode of the last command. If the ls failed it will be a number greater 0. I don't need that but some of my users can't focus on their work if they don't see the zeroes. Probably that's a shell thing. 😄

The shell is likely to be bash, though. I'm not so sure what creates these command sequences though. If I execute the same command without python directly on the shell I don't get any of that. Could it be an issue between string encodings? Because I don't use Python3 my strings might not be utf8, or they might be utf8 but some of the programs involved might expect latin1 encoding? Something like this might result in a terminal interpreting something as a command character which is actually supposed to be something completely different, right?

Because you already know so much about terminals I have some more questions:

  1. Most importantly, how did you get that knowledge? It's rather hard to find a practical guide to handle all that. Beside everything discussed here, which I would have gladly learned myself from google (which I tried first, of course) instead of requiring your time, there are also things like after a certain amount of characters something tries to \r while writing the input. It's clear why this happens. If I write a shell command that is very long, I will continue in the next line at some point. Otherwise my writing would be outside of the terminal at some point. I wasn't able to deactivate that or at least increase the length of the string before it happens by changing the terminal width and I feel unable to find out why. It would help a great deal if you could share some sources to educate myself.
  2. Is my assumption correct about the programs involved in one pxssh session:
[user] <-> [e.g. gnome terminal emulator] <-> [bash] <-> [python] <-> [pexpect] <-> [pty] <-> [/usr/bin/ssh] <==> [network] <==> [sshd] <-> [also a pseudoterminal?] <-> [bash no. 2] <-> [the actual command]

If that's really the case such a command sequence could be created by any of the programs in any direction right? Even the user could enter something like \x1b[K into its stream, and the sshd side would echo it back to show that it was typed by the user. The same goes for the echo=False. Something still prints the input, even if echo=False. Actually if I echo=False and then echo=True I get it back twice.

@takluyver
Copy link
Member

Here's a handy reference about ttys (which covers ptys): http://www.linusakesson.net/programming/tty/index.php

I'm working on separating the 'wait for output' components of Pexpect more neatly from the 'pty management' parts, so that it's easier to use one without the other. See e.g. #123.

@jquast
Copy link
Member

jquast commented Nov 25, 2014

If I execute the same command without python directly on the shell I don't get any of that

You do. Try this:

$ script
Script started, output file is typescript
$ ls
blah blah blah
$ exit
Script done, output file is typescript
$ less typescript

For me, I see for example:

Script started on Mon Nov 24 22:14:54 2014
ESC[?1034h$ ls
blah blah blah

The reason you do not see it is because your terminal emulator interprets them.

Is there a way to tell a terminal to recognize them, consume them, but not do what they are telling the terminal to do, i.e., not to move up once, not move to the right and not move to column 0?

This is what the blessed example does -- it recognizes them, but does not "do" them, only "strips" them.

Could it be an issue between string encodings?

No, terminal sequences are unrelated to encodings.

Because I don't use Python3 my strings might not be utf8, or they might be utf8 but some of the programs involved might expect latin1 encoding?

latin1 and utf8 apply the same between python 2 and 3.

In python 3, you must explicitly write code to translate between raw bytes and encodings, where in python2 it is very easy to coerce them (often as 'ascii', raising UnicodeDecodeError).

For running remote ssh commands, the remote system will use whatever encoding is specified by the LANG variable that you export.

Because you already know so much about terminals I have some more questions:

  1. Most importantly, how did you get that knowledge?

Mostly from Richard Steven's Advance Unix Programming (AUP) book, http://www.amazon.com/exec/obidos/ISBN=0201563177/wrichardstevensA/

Chapters 11. Terminal I/O and 19. Pseudo Terminals

If that's really the case such a command sequence could be created by any of the programs in any direction right?

Only "bash #2" and "the actual command" would do this.

[sshd] <-> [also a pseudoterminal?]

Yes, also a pseudo-terminal.

Even the user could enter something like \x1b[K into its stream

\x1b[K is move to beginning of line, it is not an input sequence. It just so happens that the output sequence for up, \x1b[A is also many terminal emulator's input sequences for up, \x1b[A.

Something still prints the input, even if echo=False.

Well, if I press escape, then [, then K in bash, i'll get a bell (\x07). If I press escape, then [, then A in bash, I'll get the previous command (up arrow).

Actually if I echo=False and then echo=True I get it back twice.


anyway, I understand this issue to remain now as Q: "I would like to use pexpect's interface, without a pseudo-tty, perhaps something provided by paramiko, or subprocess.Popen()".

A: This is possible using pexpect.fdexpect, http://pexpect.readthedocs.org/en/3.x/api/fdpexpect.html

If you find any bugs, please open an issue. Otherwise stack overflow might be your best avenue from here.

@jquast jquast closed this as completed Nov 25, 2014
@jquast
Copy link
Member

jquast commented Nov 25, 2014

One final note, regarding "getting double output even with echo=False", try issuing a command on the remote system to disable echo as part of your pexpect script:

stty -echo

@erikbgithub
Copy link
Contributor Author

Thanks guys. You gave me a lot to read. Don't underestimate your knowledge, though. If I hadn't unsuccessfully googled on that topic for months and got no valuable feedback on SO I wouldn't have come here. I can't appreciate your help enough!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants