need a way to redirect STDOUT/ERR/IN to Julia streams

[stevengj]

In order to implement the IPython front-end (see e.g. julia-ipython issue #5), we need to be able to capture stdout and stderr and redirect them to messages periodically flushed to a ZMQ socket (to send to the IPython front-end). We also need to capture stdin read requests and turn them in to requests over ZMQ to the IPython front-end.

We can, of course, temporarily modify the variables STDIN etcetera (although even this seems to require some changes to Base, since Julia doesn't like it when code tries to change globals owned by another module). But it would be much nicer to capture the actual file descriptors, e.g. so that we can capture output from external C libraries.

@JeffBezanson suggested doing some sort of dup on the file descriptors (though this may be harder on Windows). @loladiro, any thoughts?

[Keno]

Yeah, I have an idea. Hold on.

[Keno]

Hope you don't mind using lots of unexported API:

julia> read, write = Base.NamedPipe(C_NULL), Base.NamedPipe(C_NULL)
(NamedPipe(null, 0 bytes waiting),NamedPipe(null, 0 bytes waiting))

julia> Base.link_pipe(read,true,write,false)

julia> eval(Base,:(STDOUT = $write))
NamedPipe(open, 0 bytes waiting)

julia> println("Hello World")

julia> readline(read)
"Hello World\n"

Note that you won't be able to test this with the default readline REPL, as that just writes to STDOUT. However, if you grab the latest master of my REPL, Readline and Terminals packages and run this script:

using REPL
using Terminals
if length(ARGS) == 1
    include(ARGS[1])
else
    REPL.run_repl(Terminals.Unix.UnixTerminal("xterm",STDIN,STDOUT,STDERR))
end

you should be able to play with this more easily, since it grabs the original TTY streams.

[Keno]

After that, we might also want to actually dup the generated file descriptors onto the default STDIN, STDOUT, STDERR, etc. but that might a thing to do later. This will capture all output from julia and any external processes launched by it.

[Keno]

Ok, I have tested this on linux. Should work on windows as well:

using REPL 
using Terminals
@unix_only dup(x::RawFD) = RawFD(ccall(:dup,Int32,(Int32,),x.fd))
@windows_only dup(x::RawFD) = RawFD(ccall(:_dup,Int32,(Int32,),x.fd))
@unix_only dup(src::RawFD,target::RawFD) = systemerror("dup",ccall(:dup2,Int32,(Int32,Int32),src.fd,target.fd) == -1)
import Base: NamedPipe, fd, TTY
@unix_only fd(x::NamedPipe) = RawFD(ccall(:jl_uv_pipe_fd,Int32,(Ptr{Void},),x.handle))
@windows_only fd(x::NamedPipe) = WindowsRawSocket(ccall(:jl_uv_pipe_handle,Ptr{Void},(Ptr{Void},),x.handle))

old_in = dup(RawFD(0))
old_out = dup(RawFD(1))
old_err = dup(RawFD(2))

stdin_read, stdin_write = (Base.NamedPipe(C_NULL), Base.NamedPipe(C_NULL))
stdout_read, stdout_write = (Base.NamedPipe(C_NULL), Base.NamedPipe(C_NULL))
stderr_read, stderr_write = (Base.NamedPipe(C_NULL), Base.NamedPipe(C_NULL))

Base.link_pipe(stdin_read,false,stdin_write,true)
Base.link_pipe(stdout_read,true,stdout_write,false)
Base.link_pipe(stderr_read,true,stderr_write,false)

@unix_only begin
    dup(fd(stdin_read),   RawFD(0))
    dup(fd(stdout_write),  RawFD(1))
    dup(fd(stderr_write),  RawFD(2))
end

@windows_only begin
    ccall(:SetStdHandle,stdcall,Int32,(Uint32,Ptr{Void}),-10,fd(stdin_read).handle)
    ccall(:SetStdHandle,stdcall,Int32,(Uint32,Ptr{Void}),-11,fd(stdout_read).handle)
    ccall(:SetStdHandle,stdcall,Int32,(Uint32,Ptr{Void}),-12,fd(stderr_read).handle)
end

eval(Base,quote
    STDIN = $stdin_read 
    STDOUT = $stdout_write
    STDERR = $stderr_write
end)

if length(ARGS) == 1
    include(ARGS[1])
else
    REPL.run_repl(Terminals.Unix.UnixTerminal("xterm",TTY(old_in;readable=true),TTY(old_out),TTY(old_err)))
end

[stevengj]

Great! @JeffBezanson and @StefanKarpinski, should we put something like this directly into Base or keep it in julia-ipython?

[stevengj]

By "like this", I mean a standard API for interposing a custom IOStream in front of stdout etc.

[JeffBezanson]

Awesome, thanks!

Since this uses dup, isn't it unnecessary to reassign STDOUT etc. in Base?

Can we have an API that looks more like:

stdout_read, stdout_write = pipeends()
dup(stdout_write, STDOUT)

[Keno]

No, we still need to assign STDOUT, since it's not a TTY anymore.

[Keno]

Plus windows doesn't actually rely on the file descriptors, but on the handle that was assigned to init on startup.

[JeffBezanson]

The TTY distinction continues to annoy I see. In this case, that libuv detail seems to entirely prevent us from having an API like dup(pipe_end, STDOUT). I guess we can settle for stdout_read, stdout_write = stdout_redirect() or something like that.

[Keno]

Implemented as proposed by @JeffBezanson above:

read, write = stdout_redirect()

[stevengj]

What do we do if we still want to write some things to the original stdout? (e.g. logging messages?)

[Keno]

You'll have to dup the original file descriptors. I can add convenience methods that do this.

[stevengj]

Convenience methods to save IO streams for the original file descriptors would be great, thanks.

[vtjnash]

@loladiro instead of un-reverting your hack, this should be rewritten as a unconditional dup of the file descriptors at startup (so the originals are never affected), and writen in C. There's no reason for dup() to exist in the julia base library. Please use pull requests for major additions / changes, not immediately applying them on master, and not just before we want to tag a release.

[JeffBezanson]

dup is only internal. There are plenty of internal functions it's not safe to call. Could you clarify what exactly the problem is?

[vtjnash]

It hacks into libuv in ways that are unnecessary and potentially unsafe

[Keno]

I don't mind duping the file descriptor on startup. I was gonna do that anyway. That doesn't solve the issue of having the _fd or dup functions though, which I think are fine to have as long as they are used strictly internally.

[Keno]

I do apologize for not pull-requesting this though. I probably should have done that.

[Keno]

@stevengj No need to call any save functions now

old_stdout = STDOUT

before redirecting will do the trick.

[vtjnash]

The point is, if you are doing the dup at startup, then it should be done in C, before creating the libuv objects. That means you never need to close or change the object libuv originally opened for stdout. It also should mean that you don't need to hack libuv to get the handles.

Can you revert the jl_uv_pipe_fd stuff again? Or does link pipe not actually expose the handle? I was thinking it did.

I like being able to comment on pull request before they affect master. It reduces the urgency of making fixes, and the number of patches.

[Keno]

link_pipe does not expose the handle.

[stevengj]

Julia STDOUT is redirected, but I/O from external C code doesn't seem to be handled. For example, the following program

const stdout = STDOUT
const rd, wr = redirect_stdout()

@async while true
    s = readavailable(rd)
    print(stdout, "STDOUT echo $(length(s)) chars: ", s)
end

println("Hello world!")
n = ccall(:printf, Cint, (Ptr{Uint8},), "Hello world again!")
println(stdout, "printf => $n \n")
wait()

gives the output

STDOUT echo 13 chars: Hello world!
printf => 18 

In other words, printf thinks that it printed 18 characters but readavailable got nothing. Or maybe we just need a way to fflush(stdout) from Julia?

[stevengj]

Nevermind, looks like it is just the fact that libc is not flushing stdout. (Calling printf in a loop eventually flushes stdout, and the output shows up.)

[stevengj]

See also #3949, which is needed to easily flush C's stdout from Julia.

[randy3k]

I discovered something strange. When I run

const stdout = STDOUT
const rd, wr = redirect_stdout()

@async while true
    s = readavailable(rd)
    print(stdout, "STDOUT echo $(length(s)) chars: \n")
end

println("Hello world!")
ccall(:printf, Cint, (Ptr{Uint8},), "Hello world again!\n")

The async task will not do anything unless I do flush_cstdio(), which is consistent with @stevengj .

HOWEVER, if I write something to STDOUT fist

ccall(:printf, Int, (Ptr{Uint8},), "bar\n");
const stdout = STDOUT
const rd, wr = redirect_stdout()

@async while true
    s = readavailable(rd)
    print(stdout, "STDOUT echo $(length(s)) chars: \n")
end

println("Hello world!")
ccall(:printf, Cint, (Ptr{Uint8},), "Hello world again!\n")

then the last line will trigger the async task and print "STDOUT.....", even without flushing.

[damiendr]

I've often wished for something like this:

out = IOBuffer()
err = IOBuffer()
open(cmd, "w", stdout=out, stderr=err) do stdin
    ...
end

Would that be possible?

[stevengj]

@damiendr, that's really a separate issue.