Skip to content
This repository

Permit kernel std* to be redirected #420

Merged
merged 4 commits into from about 3 years ago

3 participants

Evan Patterson Brian E. Granger Thomas Kluyver
Evan Patterson
Collaborator

I have added support for redirecting a launched kernel's stdin/stdout/stderr. As usual, Windows caused me some hassle.

I also fixed a few related bugs, the most significant of which was causing the heartbeat channel to poll indefinitely sometimes!

Thomas Kluyver takluyver commented on the diff
IPython/zmq/entry_point.py
((10 lines not shown))
78 81 blackhole = file(os.devnull, 'w')
79   - sys.stdout = sys.stderr = blackhole
80   - sys.__stdout__ = sys.__stderr__ = blackhole
  82 + if namespace.no_stdout:
  83 + sys.stdout = sys.__stdout__ = blackhole
6
Thomas Kluyver Collaborator
takluyver added a note

According to the docs (http://docs.python.org/library/sys.html#sys.__stdin__), the double-underscored versions of the names (sys.__stdout__ etc.) are supposed to preserve the original stdin/stdout/stderr for anything that explicitly wants to access them later. What's our rationale for overriding those here, in addition to the 'working locations' (sys.stdout etc.)?

Thomas Kluyver Collaborator
takluyver added a note

P.S. I recognise that we were already doing that before this pull request. I'm just interested to understand it.

Evan Patterson Collaborator
epatters added a note

The rational is that the sys.__stdout__ and sys.__stderr__ created for pythonw.exe are broken. If you write more than 4096 bytes to them, the interpreter will crash. Nice, huh? There is no reason to preserve them--you don't want them.

Thomas Kluyver Collaborator
takluyver added a note

...In one go? Or as soon as you get to 4096 bytes, it crashes?

So does this bit of the code only get executed under pythonw.exe? I see the check for the executable file has gone, is one of the conditionals equivalent?

Evan Patterson Collaborator
epatters added a note

As soon as you hit 4096 bytes. See http://bugs.python.org/issue706263 for more information about the bug.

This bit of code is only executed under pythonw.exe and only when the streams are not explictly redirected by the kernel launch process. The latter condition is the reason that I can't leave the check for pythonw.exe in the kernel process itself: there is no way, as far as I can tell, for the kernel process to know if it has a '"bad" stdout or stderr. I am using the --no-stdout and --no-stderr options to communicate that information.

Evan Patterson Collaborator
epatters added a note

To be clear: the reason that I was able to do the check for pythonw.exe in the kernel process before was that there was no option to redirect the kernel's stdout or stderr. I could do whatever I wanted with them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Brian E. Granger
Owner

Just to clarify the need for this. Isn't the stdout/stderr of the kernel redirected to the zmq channel? What type of redirect is this then?

Evan Patterson
Collaborator

It is redirected to the ZMQ channel if everything is set up correctly.

The use case that I have for this is as follows: my application launches a kernel in a way that should be transparent to the user. However, the kernel may crash while launching if, say, an unavailable matplotlib backend has been selected. In this case, the application needs to inform the user about the crash in a way that is actually useful. To do this, the exception message produced by the kernel must be captured.

Brian E. Granger
Owner

Ahh, OK, that really helps. So this is to cover a edge case that itself is exceptional.

Evan Patterson
Collaborator

Right. Is this ready to merge?

Brian E. Granger
Owner
Evan Patterson epatters merged commit 8879791 into from
Evan Patterson epatters closed this
Brian E. Granger ellisonbg referenced this pull request from a commit
Commit has since been removed from the repository and is no longer available.
Damián Avila damianavila referenced this pull request from a commit
Commit has since been removed from the repository and is no longer available.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
72 IPython/zmq/entry_point.py
@@ -23,6 +23,7 @@
23 23 from parentpoller import ParentPollerUnix, ParentPollerWindows
24 24 from session import Session
25 25
  26 +
26 27 def bind_port(socket, ip, port):
27 28 """ Binds the specified ZMQ socket. If the port is zero, a random port is
28 29 chosen. Returns the port that was bound.
@@ -51,6 +52,10 @@ def make_argument_parser():
51 52 help='set the REQ channel port [default: random]')
52 53 parser.add_argument('--hb', type=int, metavar='PORT', default=0,
53 54 help='set the heartbeat port [default: random]')
  55 + parser.add_argument('--no-stdout', action='store_true',
  56 + help='redirect stdout to the null device')
  57 + parser.add_argument('--no-stderr', action='store_true',
  58 + help='redirect stderr to the null device')
54 59
55 60 if sys.platform == 'win32':
56 61 parser.add_argument('--interrupt', type=int, metavar='HANDLE',
@@ -71,13 +76,13 @@ def make_kernel(namespace, kernel_factory,
71 76 """ Creates a kernel, redirects stdout/stderr, and installs a display hook
72 77 and exception handler.
73 78 """
74   - # If running under pythonw.exe, the interpreter will crash if more than 4KB
75   - # of data is written to stdout or stderr. This is a bug that has been with
76   - # Python for a very long time; see http://bugs.python.org/issue706263.
77   - if sys.executable.endswith('pythonw.exe'):
  79 + # Re-direct stdout/stderr, if necessary.
  80 + if namespace.no_stdout or namespace.no_stderr:
78 81 blackhole = file(os.devnull, 'w')
79   - sys.stdout = sys.stderr = blackhole
80   - sys.__stdout__ = sys.__stderr__ = blackhole
  82 + if namespace.no_stdout:
  83 + sys.stdout = sys.__stdout__ = blackhole
  84 + if namespace.no_stderr:
  85 + sys.stderr = sys.__stderr__ = blackhole
81 86
82 87 # Install minimal exception handling
83 88 sys.excepthook = FormattedTB(mode='Verbose', color_scheme='NoColor',
@@ -155,6 +160,7 @@ def main():
155 160
156 161
157 162 def base_launch_kernel(code, xrep_port=0, pub_port=0, req_port=0, hb_port=0,
  163 + stdin=None, stdout=None, stderr=None,
158 164 executable=None, independent=False, extra_arguments=[]):
159 165 """ Launches a localhost kernel, binding to the specified ports.
160 166
@@ -175,6 +181,9 @@ def base_launch_kernel(code, xrep_port=0, pub_port=0, req_port=0, hb_port=0,
175 181 hb_port : int, optional
176 182 The port to use for the hearbeat REP channel.
177 183
  184 + stdin, stdout, stderr : optional (default None)
  185 + Standards streams, as defined in subprocess.Popen.
  186 +
178 187 executable : str, optional (default sys.executable)
179 188 The Python executable to use for the kernel process.
180 189
@@ -228,15 +237,35 @@ def base_launch_kernel(code, xrep_port=0, pub_port=0, req_port=0, hb_port=0,
228 237 interrupt_event = ParentPollerWindows.create_interrupt_event()
229 238 arguments += [ '--interrupt', str(int(interrupt_event)) ]
230 239
231   - # If using pythonw, stdin, stdout, and stderr are invalid. Popen will
232   - # fail unless they are suitably redirected. We don't read from the
233   - # pipes, but they must exist.
234   - redirect = PIPE if executable.endswith('pythonw.exe') else None
235   -
  240 + # If this process in running on pythonw, stdin, stdout, and stderr are
  241 + # invalid. Popen will fail unless they are suitably redirected. We don't
  242 + # read from the pipes, but they must exist.
  243 + if sys.executable.endswith('pythonw.exe'):
  244 + redirect = True
  245 + _stdin = PIPE if stdin is None else stdin
  246 + _stdout = PIPE if stdout is None else stdout
  247 + _stderr = PIPE if stderr is None else stderr
  248 + else:
  249 + redirect = False
  250 + _stdin, _stdout, _stderr = stdin, stdout, stderr
  251 +
  252 + # If the kernel is running on pythonw and stdout/stderr are not been
  253 + # re-directed, it will crash when more than 4KB of data is written to
  254 + # stdout or stderr. This is a bug that has been with Python for a very
  255 + # long time; see http://bugs.python.org/issue706263.
  256 + # A cleaner solution to this problem would be to pass os.devnull to
  257 + # Popen directly. Unfortunately, that does not work.
  258 + if executable.endswith('pythonw.exe'):
  259 + if stdout is None:
  260 + arguments.append('--no-stdout')
  261 + if stderr is None:
  262 + arguments.append('--no-stderr')
  263 +
  264 + # Launch the kernel process.
236 265 if independent:
237 266 proc = Popen(arguments,
238 267 creationflags=512, # CREATE_NEW_PROCESS_GROUP
239   - stdout=redirect, stderr=redirect, stdin=redirect)
  268 + stdin=_stdin, stdout=_stdout, stderr=_stderr)
240 269 else:
241 270 from _subprocess import DuplicateHandle, GetCurrentProcess, \
242 271 DUPLICATE_SAME_ACCESS
@@ -245,21 +274,26 @@ def base_launch_kernel(code, xrep_port=0, pub_port=0, req_port=0, hb_port=0,
245 274 True, # Inheritable by new processes.
246 275 DUPLICATE_SAME_ACCESS)
247 276 proc = Popen(arguments + ['--parent', str(int(handle))],
248   - stdout=redirect, stderr=redirect, stdin=redirect)
  277 + stdin=_stdin, stdout=_stdout, stderr=_stderr)
249 278
250 279 # Attach the interrupt event to the Popen objet so it can be used later.
251 280 proc.win32_interrupt_event = interrupt_event
252 281
253 282 # Clean up pipes created to work around Popen bug.
254   - if redirect is not None:
255   - proc.stdout.close()
256   - proc.stderr.close()
257   - proc.stdin.close()
  283 + if redirect:
  284 + if stdin is None:
  285 + proc.stdin.close()
  286 + if stdout is None:
  287 + proc.stdout.close()
  288 + if stderr is None:
  289 + proc.stderr.close()
258 290
259 291 else:
260 292 if independent:
261   - proc = Popen(arguments, preexec_fn=lambda: os.setsid())
  293 + proc = Popen(arguments, preexec_fn=lambda: os.setsid(),
  294 + stdin=stdin, stdout=stdout, stderr=stderr)
262 295 else:
263   - proc = Popen(arguments + ['--parent'])
  296 + proc = Popen(arguments + ['--parent'],
  297 + stdin=stdin, stdout=stdout, stderr=stderr)
264 298
265 299 return proc, xrep_port, pub_port, req_port, hb_port
5 IPython/zmq/ipkernel.py
@@ -551,6 +551,7 @@ def start(self):
551 551 #-----------------------------------------------------------------------------
552 552
553 553 def launch_kernel(ip=None, xrep_port=0, pub_port=0, req_port=0, hb_port=0,
  554 + stdin=None, stdout=None, stderr=None,
554 555 executable=None, independent=False, pylab=False, colors=None):
555 556 """Launches a localhost kernel, binding to the specified ports.
556 557
@@ -571,6 +572,9 @@ def launch_kernel(ip=None, xrep_port=0, pub_port=0, req_port=0, hb_port=0,
571 572 hb_port : int, optional
572 573 The port to use for the hearbeat REP channel.
573 574
  575 + stdin, stdout, stderr : optional (default None)
  576 + Standards streams, as defined in subprocess.Popen.
  577 +
574 578 executable : str, optional (default sys.executable)
575 579 The Python executable to use for the kernel process.
576 580
@@ -608,6 +612,7 @@ def launch_kernel(ip=None, xrep_port=0, pub_port=0, req_port=0, hb_port=0,
608 612 extra_arguments.append(colors)
609 613 return base_launch_kernel('from IPython.zmq.ipkernel import main; main()',
610 614 xrep_port, pub_port, req_port, hb_port,
  615 + stdin, stdout, stderr,
611 616 executable, independent, extra_arguments)
612 617
613 618
14 IPython/zmq/kernelmanager.py
@@ -573,7 +573,8 @@ def run(self):
573 573 # list, poll is working correctly even if it
574 574 # returns quickly. Note: poll timeout is in
575 575 # milliseconds.
576   - self.poller.poll(1000*until_dead)
  576 + if until_dead > 0.0:
  577 + self.poller.poll(1000 * until_dead)
577 578
578 579 since_last_heartbeat = time.time()-request_time
579 580 if since_last_heartbeat > self.time_to_dead:
@@ -835,8 +836,15 @@ def kill_kernel(self):
835 836 except OSError, e:
836 837 # In Windows, we will get an Access Denied error if the process
837 838 # has already terminated. Ignore it.
838   - if not (sys.platform == 'win32' and e.winerror == 5):
839   - raise
  839 + if sys.platform == 'win32':
  840 + if e.winerror != 5:
  841 + raise
  842 + # On Unix, we may get an ESRCH error if the process has already
  843 + # terminated. Ignore it.
  844 + else:
  845 + from errno import ESRCH
  846 + if e.errno != ESRCH:
  847 + raise
840 848 self.kernel = None
841 849 else:
842 850 raise RuntimeError("Cannot kill kernel. No kernel is running!")
8 IPython/zmq/pykernel.py
@@ -248,6 +248,7 @@ def _symbol_from_context(self, context):
248 248 #-----------------------------------------------------------------------------
249 249
250 250 def launch_kernel(ip=None, xrep_port=0, pub_port=0, req_port=0, hb_port=0,
  251 + stdin=None, stdout=None, stderr=None,
251 252 executable=None, independent=False):
252 253 """ Launches a localhost kernel, binding to the specified ports.
253 254
@@ -268,6 +269,9 @@ def launch_kernel(ip=None, xrep_port=0, pub_port=0, req_port=0, hb_port=0,
268 269 hb_port : int, optional
269 270 The port to use for the hearbeat REP channel.
270 271
  272 + stdin, stdout, stderr : optional (default None)
  273 + Standards streams, as defined in subprocess.Popen.
  274 +
271 275 executable : str, optional (default sys.executable)
272 276 The Python executable to use for the kernel process.
273 277
@@ -291,8 +295,8 @@ def launch_kernel(ip=None, xrep_port=0, pub_port=0, req_port=0, hb_port=0,
291 295
292 296 return base_launch_kernel('from IPython.zmq.pykernel import main; main()',
293 297 xrep_port, pub_port, req_port, hb_port,
294   - executable, independent,
295   - extra_arguments=extra_arguments)
  298 + stdin, stdout, stderr,
  299 + executable, independent, extra_arguments)
296 300
297 301 main = make_default_main(Kernel)
298 302

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.