asynchronous messages blocking until a key is pressed

[soggysec]

I'm having an issue printing asynchronous messages while using cmdloop(). I have a CLI, and a threaded server that should be able to print messages as it receives them. On Ubuntu 16.04 x64, the asynch messages are printed as expected, but on both Debian 8 and Kali the buffer doesn't seem to flush until a key is pressed. As soon as any key is pressed, the expected messages are printed.

[tleonhardt]

@soggysec Thanks for the issue report.

If you could provide us with some additional information, it would likely be helpful. First and foremost, what exact version of Python are you using on each of the OSes you mentioned? Also, what version of readline is installed on each of those OSes?

Additionally, what command are you using to print these messages?

I've used cmd2 with both asynchronous and blocking commands extensively and have never had any problems. Depending on what your threads are doing, you may need to create an RLock (equivalent to a recursive mutex) to synchronize access to a relevant resource.

I know that @kmvanbrunt has a really nice example that may demonstrate how to do what you want which I've been encouraging him to commit ;-)

[soggysec]

@tleonhardt - Thanks for the response! I would love to see the @kmvanbrunt example, as I haven't quite ruled out the locking issue like you've described. The thing pointing me towards believing it was cmd2 is that the issue came about once I upgraded from an older version.

Python: 2.7.13

import readline
print readline._READLINE_VERSION
1792
$ dpgk -l | grep readline
libreadline7:amd64 7.0-3

As for command to print out messages, I've tried both print, and stdout.write(). There was no difference in functionality between these two.

[kmvanbrunt]

@soggysec - I'll try to get an example posted this evening.
Out of curiosity, does calling sys.stdout.flush() after your call to print() make the output appear without having to press a key?

[soggysec]

@kmvanbrunt Nope. flushing hasn't changed the result. I also tried running it with python -u, which apparently removes buffering, but it did not help either.

[soggysec]

Hey guys, here's an almost equivalent of what I'm trying to do, and what's failing. On Kali and Debian 8, but works on Ubuntu 16.04. I copied two different samples online and smashed their parts together:

Code pasted here, as this wasn't liking the codeblocks: https://pastebin.com/Lm80mqLH

[kmvanbrunt]

@soggysec
OK, I've narrowed this down to strange behavior between Python 2's raw_input() function and pygtk.

cmd2 imports the pyperclip module, which imports gtk if it exists on the system. I'm guessing that your Ubuntu system does not have Python gtk installed, so pyperclip is using a fallback method for its clipboard functionality.

For some reason, if gtk is imported, then no printing will occur when raw_input() is blocking. I do have a work around for you. After calling Cmd's initializer, set its use_rawinput member to False. This tells cmd2 to use readline directly to get its input.

Cmd.__init__(self, use_ipython=False)
self.use_rawinput = False

Below is code that demonstrates the bug without involving cmd2. As written the code will print correctly. If you uncomment the "import gtk" line then the printing will not work.

#!/usr/bin/env python
# coding=utf-8

#import gtk
import six.moves as sm
import threading
import time

stop_printing = False


def print_thread_worker():
        time_slept = 0
        SLEEP_INTERVAL = 0.2

        while not stop_printing:
            time.sleep(SLEEP_INTERVAL)
            time_slept += SLEEP_INTERVAL

            if time_slept >= 1:
                print("PRINTING")
                time_slept = 0

if __name__ == '__main__':

    print("Wait a second for the printing to start and then hit Enter to exit")

    print_thread = threading.Thread(target=print_thread_worker)
    print_thread.start()

    try:
        line = sm.input('cli> ')
    except (Exception, KeyboardInterrupt):
        pass

    stop_printing = True
    if print_thread.is_alive():
        print_thread.join()

I have no idea the root cause of this. For now, if you plan on using Python 2 with a system that has pygtk, then use the workaround I described.

[soggysec]

Wow, what a strange bug. So that seems to make my sample program work, but now my real application is bailing immediately after cmdloop.

My code looks like the following

class MyClass(Cmd):
    def __init__(self):
        self.prompt = "> "
        Cmd.__init__(self)
        self.allow_cli_args = False
        self.use_rawinput = False

as soon as cmdloop is called, it exits.

[tleonhardt]

@soggysec I tried making a minimal app like you suggested:

 #!/usr/bin/env python
from cmd2 import Cmd

class MyClass(Cmd):
    def __init__(self):
        self.prompt = "> "
        Cmd.__init__(self)
        self.allow_cli_args = False
        self.use_rawinput = False

if __name__ == '__main__':
    app = MyClass()
    app.cmdloop()

In this case the cmdloop() runs as expected. Can you share the complete example where cmdloop is exiting as soon as it is called (or a scaled-down demonstration which we can use to reproduce that issue)?

However, readline tab completion is totally broken. We will look into this issue some more and see if we can't find a better workaround for you.

As @kmvanbrunt mentioned in his excellent writeup regarding what is going wrong, this fundamentally isn't a bug in cmd2. The reason you didn't see this in older versions of cmd2 is that we only recently added the dependency on pyperclip, which in turn has the optional dependency on pygtk. It appears to be pygtk's setting of interactive mode which is messing up output.

Also, this bug does not appear to be present in Python 3. For many other reasons I highly recommend porting your code to Python 3 if you have the opportunity. The ability to use optional type hinting present in Python 3.5 and 3.6 is wonderful. Now that I have it and PyCharm's nice integration with it (as well as mypy) I'm never going back to Python 2. I look forward to the day where I can abandon support for Python 2 in cmd2.

[soggysec]

Ah, I was just going to say. I found an issue in my code that was causing it to bail, but now the tab-completion is broken. If that's as-expected, that's a bit rough. What version would you recommend to still have tab-completion, but not have the pygtk bug?

[tleonhardt]

@soggysec So the tab-completion not working when use_rawinput is False probably IS either a cmd2 bug or a cmd bug. I'll look into it and probably create a new issue for that.

If you have the option of using Python 3, then things should probably work ok for you with that and leaving the default value of True for use_rawinput. To the best of my knowledge, the pygtk bug is not present in Python 3.

@kmvanbrunt Can you confirm that the pygtk bug isn't present in Python 3?

[soggysec]

Unfortunately Python3 is a ways out for us. We are hoping to transition, but not quite ready, yet. Please let me know if you create an issue for this, so I can follow along.

I'm also seeing an extra newline coming after the prompt, but I haven't looked into it fully, and it could be an issue on my end. I'm setting self.prompt = "> ", but when it's printed, I'm getting a:

I appreciate your guys' help!

[tleonhardt]

@soggysec I created #206 to track the issue of broken tab completion when use_rawinput is False

[tleonhardt]

Just to summarize for a TLDR:

This is a bug in pygtk in Python 2 (but not Python 3) exposed by cmd2's dependency on pyperclip. It only occurs when all of the following conditions are true:

• OS = Linux
• Python version = 2.7
• pygtk is installed
• cmd2 version >= 0.7.4

[kmvanbrunt]

@soggysec
This is kind of a hack, but it appears to work and it does not require setting use_rawinput, so tab completion is not broken.

Put this after you import cmd2.

# Workaround for gtk interfering when printing from background thread while main
# thread is blocking in raw_input()
try:
    import gtk
    gtk.set_interactive(0)
except ImportError:
    pass

[tleonhardt]

@soggysec

I like the hack @kmvanbrunt recommended above. I think that is likely to be a good solution for you. We may even incorporate that into cmd2 as a proactive workaround.

Another possibility would be to run your application in an environment where pygtk isn't installed, possibly a virtualenv where you carefully control the Python modules installed.