new repl wrong STDERR stream

[vtjnash]

the new repl is incorrectly writing errors to STDOUT rather than STDERR (as would usually be expected)

$ ./julia 2> /dev/null
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "help()" to list help topics
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.3.0-prerelease+4147 (2014-07-12 04:48 UTC)
 _/ |\__'_|_|_|\__'_|  |  Commit 772f312* (4 days old master)
|__/                   |  x86_64-apple-darwin13.2.0

julia> 2^-1 # this error message should be swallowed by /dev/null
ERROR: DomainError
 in ^ at intfuncs.jl:84

julia>

[kmsquire]

Interestingly, on Linux, I don't even get the prompt:

$ ./julia 2> stderr.txt

$ cat stderr.txt
ERROR: `convert` has no method matching convert(::Type{TTY}, ::IOStream)
 in TTYTerminal at no file
 in _start at ./client.jl:363

[vtjnash]

oops, i probably should have posted exactly what I tested, rather than demonstrating a different bug

the following reproduces my original result on OSX:

$ ./julia 2> /dev/ttys001

and on Ubuntu:

$ ./julia 2> /dev/pts/28

[LaszloHars]

In Windows error messages and warnings don't appear in STDOUT. It is redirected with the commands below in the console (REPL), so if they were seen in STDOUT, they should have disappeared after redirection:

julia> rd,wr = redirect_stdout()
(Pipe(open, 0 bytes waiting),Pipe(open, 0 bytes waiting))

julia> pushdisplay(TextDisplay(wr))

julia> 1+2

julia> 1^-1
ERROR: DomainError
 in power_by_squaring at intfuncs.jl:60

julia> warn("test")
WARNING: test

Similar experiments with STDERR:

julia> rd,wr = redirect_stderr()

julia> pushdisplay(TextDisplay(wr))

julia> warn("test")

julia> 1^-1
ERROR: DomainError
 in power_by_squaring at intfuncs.jl:60

The warning is not shown in the console anymore, indicating that it is sent to STDERR. If it is the correct stream for warnings is debatable. However, the error message is still shown in the console, and It is put neither in STDOUT nor in STDERR, just got displayed in the console window.

If it is the intended behavior, the STDERR stream is superfluous, as error messages do not appear in there.

[stevengj]

@LaszloHars, we are discussing a separate question from what file descriptors are used for side effects from print, warn, etc. in Julia code. The question here is about a property of the REPL program, not about the Julia language and standard library.

The question raised by @vtjnash is what file descriptor the REPL should ultimately write its output to, and in particular whether it should use different file descriptors for writing the result of evaluated expressions vs. exceptions caught from those expressions.

[stevengj]

I'm not actually convinced that this is a bug. Why should the REPL write its exception output to stderr? If I write error("foo"), then catching the exception and displaying the backtrace is what the REPL is supposed to do, not an error, so arguably it belongs on stdout.

To make an analogy, if you run gdb on a program and the program segfaults, then gdb prints a stack trace. I don't think that gdb prints the stack trace on stderr, nor should it: the stack trace is not an error in gdb.

[stevengj]

In fact, my inclination would be to go in the opposite direction:

• Make an ExceptionData type that stores an (Exception, backtrace) pair. Define a show method which pretty-prints this by calling showerror.
• Have the REPL display exceptions by calling display(ExceptionData(err, backtrace)).

This way, if you have some fancier way you want to display exceptions in the REPL, you can push a corresponding display onto the display stack. But it would go to standard output by default.

[LaszloHars]

I was told in a julia-users Google group conversation about STDERR that this was the relevant issue in github. Sorry if it was not the case. Anyway, @Steve:

redirect_stdout has no effect
redirect_stdout does work, as I just demonstrated. If it should not, then it is a bug.

[an error message] belongs on stdout
but error messages are not written to STDOUT, either, as I just demonstrated. If they should be, then it is a bug.

If the REPL does not write error to STDERR, then what is the purpose of this stream? The documentation ought to mention that STDIN, STDOUT and STDERR must not be used from the REPL, otherwise naive users (like me) spend days struggling with them.

[stevengj]

redirect_stdout doesn't redirect things like the julia> prompt, which is also written to the standard output file descriptor. Nor should it. The point is that redirect_stdout is mainly for redirecting side effects from running Julia code, not side effects from the REPL.

And you can use STDERR from the REPL. That's separate from the question of whether and how the REPL itself uses STDERR.

And part of the reason you've been struggling is that you've been spending nearly a month ignoring the repeated advice of experienced developers on the mailing list.

[LaszloHars]

I really like the idea of ExceptionData. It would better solve my problems than redirecting STD streams. Could we expect changes in this direction in the near future? (Still, the documentation ought to say something about STD streams and the REPL.)

[LaszloHars]

"Ignoring the repeated advice of experienced developers on the mailing list"

Almost all of the "repeated advice" was about something else than the question at hand. I asked, how to capture the REPL output. The pieces of advice I got were "do this instead of capturing the output". I did not need those pieces of advice, since I had working "hacks". As I learned, the simple answer to my question: "you cannot do that with code of reasonable complexity".
Now I could start other discussions (but I would not) about the three or four different applications I had which would benefit from capturing the REPL output. I just wait till ExceptionData will be available.

[vtjnash]

Make an ExceptionData type that stores an (Exception, backtrace) pair. Define a show method which pretty-prints this by calling showerror.

A rather long time ago (in Julia), we had discussed various ways of improving backtrace handling in Exceptions. Proposals included adding a backtrace to every Exception, or making an ExceptionData type. I guess we didn't have enough motivation at the time to actually try implementing some of these ideas. One of the problems is that generating a backtrace is extremely expensive – perhaps 99% of the work of throwing an exception – so it would be nice to be able to avoid creating one when it's not used

I'm not actually convinced that this is a bug. Why should the REPL write its exception output to stderr?

I don't have an answer. My best attempt at an answer is "that's what I expected". It seems inconsistent otherwise (since the stream used to print the error depends on whether it is caught by the REPL, or one of the other error handler, such as the client loop or root task unhandled exception handler). It's also what python does:

$ python 2> /dev/null 
>>> 
>>> xx
>>> ^D
$ python 
Python 2.7.5 (default, Mar  9 2014, 22:15:05) 
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> xx
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'xx' is not defined
>>> ^D

And you can use STDERR from the REPL. That's separate from the question of whether and how the REPL itself uses STDERR.

I tried to use case in some of my answers to differentiate between Base.STDERR and REPL.stderr_stream, but it probably wasn't very clear. When the julia application starts the built-in REPL program, the REPL stderr_stream is initialized to equal Base.STDERR. However, there's no reason that these relationship needs to be true, or needs to be maintained (and as Steve and I have pointed out, there are good reason for not wanting to maintain this relationship). I don't know of another language that actually provides as much convenience for implementing a REPL in the language itself, such that you can easily and sanely describe having interactions with multiple independent REPL loops (maybe Scheme?).

Almost all of the "repeated advice" was about something else than the question at hand. I asked, how to capture the REPL output. The pieces of advice I got were "do this instead of capturing the output". I did not need those pieces of advice, since I had working "hacks"

Julia developers on the mailing list tend to try to answer questions of the form "how can I solve problem X by doing Y" by telling you why doing Z is better at solving problem X then using Y. If your hack was working, you wouldn't have needed to spend the past month asking for functionality to be removed from the REPL so that your hack from 0.2 would still work in 0.3.

[stevengj]

@vtjnash, my suggestion is that the REPL would only create the ExceptionData instance when it is actually ready to display the exception, in which case it is computing the backtrace anyway, so there would be no extra overhead.

(You could also make show(io, x::Exception) compute the backtrace, so that the backtrace is only computed when you try to show it, but I'm not convinced that is a good idea; you can create Exception objects without throwing exceptions, after all.)

[stevengj]

Note that, in principle, what stream the REPL displays its exceptions to is orthogonal to whether we use display(ExceptionData(...)) (or similar) to allow exception/backtrace display to be customized. If we wanted to display some things to STDERR by default, I suppose that we could modify Base.TextDisplay to wrap two streams (io=STDOUT and errio=STDERR) instead of one (io=STDOUT) and to use the latter (errio) for printing ExceptionData objects.

But I'm still skeptical that this should go to STDERR. Exceptions caught and displayed by the REPL are different, in my mind, from exceptions caught by other fallback exception handlers.

[vtjnash]

@vtjnash, my suggestion is that the REPL would only create the ExceptionData instance when it is actually ready to display the exception, in which case it is computing the backtrace anyway, so there would be no extra overhead.

My statement was generally orthogonal also, since you want the catch_backtrace(), not the current backtrace(). For example, it might be a good idea to add the following syntax and deprecate catch_backtrace entirely, so that the syntax directly indicates whether the user is going to use the backtrace:

try
  error()
catch err, bt
  display(ExceptionData(err, bt))
end

It gets a little bit trickier with rethrow() and it's cousin finally, since it is much less obvious which backtrace the user wanted (the real answer probably being that they want all of them).

But I digress from this issue. I'm also skeptical that these should go to STDERR, so I've added the decision tag. The question is whether this was an intended change in behavior with respect to the 0.2 release that we want in the 0.3 release, or if we didn't want this behavior to change, but simply hadn't noticed/tested for it.

[vtjnash]

https://gist.github.com/vtjnash/9cd53434cc7f7e2f7a29

please comment/edit the gist if this sounds reasonable (since it would be a breaking change) and I can create another issue Julep here

[stevengj]

@vtjnash, the gist seems good to me.

[JeffBezanson]

The backtrace thing is a separate issue, feel free to open one. See also #7283.

We also have a "stream repl", used when reading from a pipe, implemented with a small bit of code in client.jl. First, it would be nice to get rid of this and only have one REPL (the new REPL implements functions with the same names, such as eval_user_input). Second, the stream REPL prints errors to STDERR. This seems more reasonable in the case of piped input, but I'm not sure.

[LaszloHars]

the stream REPL prints errors to STDERR

OMG! This was what I always needed, and nobody mentioned it in the 3 julia-users Google group conversations, where I asked for help. PLEASE don't remove it, but tell, how to use it!

[JeffBezanson]

It gets used for example by cat | julia. See client.jl:389 if you'd like to copy the code.

[jakebolewski]

cat | julia >>

[JeffBezanson]

What needs to be done here?

[vtjnash]

nothing?

[JeffBezanson]

Seems that way to me?