Backgrounded Tasks not Allowed? (but easy to slip by . . .) #457

Closed
meawoppl opened this Issue May 19, 2011 · 8 comments

Projects

None yet

5 participants

@meawoppl

I used to do something like the following in iPython all the time:
In [8]: !emacs scoal-basic.py &
OSError Traceback (most recent call last)
/home/meawoppl/Dropbox/austin-research/mscoal/ in ()
----> 1 get_ipython().system(u"emacs scoal-basic.py &")

/usr/local/lib/python2.6/dist-packages/IPython/core/interactiveshell.pyc in system(self, cmd)
1886 # os.system() if they really want a background process.

1887 if cmd.endswith('&'):
-> 1888 raise OSError("Background processes not supported.")
1889
1890 return system(self.var_expand(cmd, depth=2))

OSError: Background processes not supported.

Being the silly fellow I am, I discovered the following "workarounds"

In [9]: !emacs scoal-basic.py &
(Above has an added space at the end and seems to work fine . . .)

In [10]: !emacs scoal-basic.py & true;
(this works too . . .)

Is there some reason I should not be doing this?

@takluyver
Member

The complete comment says:

# We do not support backgrounding processes because we either use
# pexpect or pipes to read from.  Users can always just call
# os.system() if they really want a background process.

From my limited testing, it doesn't seem to have a problem with background processes, except that output printed later by the process doesn't get displayed anywhere. @fperez: You wrote this bit, can you throw any more light on the matter?

@minrk
Member
minrk commented May 19, 2011

This is related to #297. I think pexpect helps a lot in the two-process model (Qt, etc.), but it adds inappropriate restrictions in a single-process terminal.

For @meawoppl, the simplest workaround for this is actually:
get_ipython().system = os.system

Then you've skipped right over all the pexpect stuff, and !stuff simply does os.system("stuff"), with all the freedom that provides.

@meawoppl

Perhaps we could add a piece of syntax like !!, which indicates to just run this via os.system(), and that the output is not important. From my brief triage of the situation, the ! indicator is really not a shell escape any more . . . which is its more than a bit misleading . . .

@minrk
Member
minrk commented May 19, 2011

No, this needs to be fixed in the Terminal. ! should do os.system, and background processes should work just fine. This is already listed as a critical bug, blocking 0.11 (#297).

Also, '!!' already exists (it does getoutput).

@ellisonbg
Member

IIRC, this was disabled because of some very subtle issues with how
subprocesses are handles in python. I don't remember the details, but
fernando may.

Cheers,

Brian

On Thu, May 19, 2011 at 11:42 AM, meawoppl
reply@reply.github.com
wrote:

I used to do something like the following in iPython all the time:

In [8]: !emacs scoal-basic.py &

OSError                                   Traceback (most recent call last)
/home/meawoppl/Dropbox/austin-research/mscoal/ in ()
----> 1 get_ipython().system(u"emacs scoal-basic.py &")

/usr/local/lib/python2.6/dist-packages/IPython/core/interactiveshell.pyc in system(self, cmd)
  1886         # os.system() if they really want a background process.

  1887         if cmd.endswith('&'):
-> 1888             raise OSError("Background processes not supported.")
  1889
  1890         return system(self.var_expand(cmd, depth=2))

OSError: Background processes not supported.

Being the silly fellow I am, I discovered the following "workarounds"

In [9]: !emacs scoal-basic.py &
(Above has an added space at the end and seems to work fine . . .)

In [10]: !emacs scoal-basic.py & true;
(this works too . . .)

Is there some reason I should not be doing this?

Reply to this email directly or view it on GitHub:
#457

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

@minrk
Member
minrk commented May 19, 2011

There may have been a good reason for it, but it was actually @fperez that asked me to open #297. There probably are subtle issues with using os.system for !, but with the current use of pexpect, all kinds of perfectly ordinary shell actions simply don't work anymore, or (worse) only half-work.

For instance, try using !emacs (or !nano, or any program that expects any kind of meta-key input). ctrl-commands are not relayed to the subprocess. That means !emacs is completely useless. Even !nano doesn't work, because you can't exit the program properly or save the file, and ctrl-C will always terminate the process, regardless of what it is supposed to do.

The result is, as @meawoppl states, ! is no longer a shell escape. It's something else, that behaves similar to a shell escape, but not as expected in plenty of mundane cases. This is acceptable in the qtconsole, where you don't have a real terminal, and shouldn't expect perfect terminal behavior, but not in the basic terminal case.

Related to this, when using the QtConsole (or other frontends that don't forward input to the subprocess), we should make sure the subprocess shell is 'dumb', so it won't expect input that can never come.

@ellisonbg
Member

OK, thanks for the background. I didn't know that the shell escape
was that broken. We defintely need to fix this and moving away from
pexpect for this may make sense.

On Thu, May 19, 2011 at 4:21 PM, minrk
reply@reply.github.com
wrote:

There may have been a good reason for it, but it was actually @fperez that asked me to open #297.  There probably are subtle issues with using os.system for !, but with the current use of pexpect, all kinds of perfectly ordinary shell actions simply don't work anymore, or (worse) only half-work.

For instance, try using !emacs (or !nano, or any program that expects any kind of meta-key input).  ctrl-commands are not relayed to the subprocess.  That means !emacs is completely useless.  Even !nano doesn't work, because you can't exit the program properly or save the file, and ctrl-C will always terminate the process, regardless of what it is supposed to do.

The result is, as @meawoppl states, ! is no longer a shell escape.  It's something else, that behaves similar to a shell escape, but not as expected in plenty of mundane cases.  This is acceptable in the qtconsole, where you don't have a real terminal, and shouldn't expect perfect terminal behavior, but not in the basic terminal case.

Related to this, when using the QtConsole (or other frontends that don't forward input to the subprocess), we should make sure the subprocess shell is 'dumb', so it won't expect input that can never come.

Reply to this email directly or view it on GitHub:
#457 (comment)

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

@fperez
Member
fperez commented May 25, 2011

Yes, this was simply an oversight of mine. When we did the Qt console work I rewrote the system call code for pexpect, not realizing that in the single-process case we should not use pexpect. It took me a while to notice how bad the breakage was (this is super hard to test automatically, because it requires firing a subprocess with interactive input, not something that's easy to put in a test suite). We'll most certainly have this fixed up before 0.11 is out, otherwise the crowds will come after us pitchforks in hand...

@minrk minrk added a commit that closed this issue May 27, 2011
@minrk minrk split shell.system into shell.system_raw/system_piped
system_raw calls os.system, system_piped uses pexpect/utils.platform magic

use system_raw in Terminal except on Windows and in tests
use system_piped elsewhere

closes gh-297
closes gh-457
bdb012c
@minrk minrk closed this in bdb012c May 27, 2011
@mattvonrocketstein mattvonrocketstein pushed a commit to mattvonrocketstein/ipython that referenced this issue Nov 3, 2014
@minrk minrk split shell.system into shell.system_raw/system_piped
system_raw calls os.system, system_piped uses pexpect/utils.platform magic

use system_raw in Terminal except on Windows and in tests
use system_piped elsewhere

closes gh-297
closes gh-457
421c858
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment