RFC: process redirections

[carlobaldassi]

This addresses #307 (and does a little more). There are 2 commits:

1. in the first one, I added file redirections. This allows things as:

run(`ls` > "out.txt") # write to stdout
run(`ls` >> "out.txt") # append to stdout
run(`ls nonexistent.txt` .> "err.txt") # write to stderr
run(`ls nonexistent.txt` .>> "err.txt") # append to stderr
run("in.txt" > `grep xyz`) # read from stdin

Notice that I had to use .> for stderr, since 2> is obviously unavailable. I also had to add .>> to the parser. This is as close I could get to the familiar shell syntax without introducing too many new operators, but if anyone comes up with a nicer alternative it would be great.

All forms have a mirror counterpart, e.g.

run(`grep xyz` < "in.txt")

and can be used together, since the result of (>)(::Cmd,::String) is a Cmd, although they require brackets:

run((`ls a b` > "out.txt") .> "err.txt")

which of course is a side-effect of using comparison operators and inheriting their precedence rules (one more reason to think of some alternative notation, if possible).

Everything works the same when using IOStreams instead of Strings:

open("out.txt", "w") do f
    run(`ls` > f)
end

but of course in that case there are no append operators, since the mode is inherited from the stream.

Apart from notation issues, I think the internals are sound, as I basically extended what's already done for pipes.

2. The second commit introduces here-strings and 2-in-1 redirections:

run("one\ntwo\nthree" >>> `grep t`) # prints two and three
run(`ls a b` &> "both.txt") # redirects both stdout and stderr
run(`ls a b` &>> "both.txt") # same as above, append mode

There are also the mirror versions <<<, &< and &<<

So, even more extra operators...

[JeffBezanson]

Looks really cool! @StefanKarpinski will have to review.

[StefanKarpinski]

Superficial review: looks great. I will take a deeper look at the code a bit later. I especially enjoy that relatively few and simple changes were made to allow this (which I guess says that the original design was not terrible). Beautiful work,@carlobaldassi . @JeffBezanson, is there anything we can do about operator precedence to remove the need for parens here without messing up normal comparisons?

[JeffBezanson]

Not really. One could try to exploit the call to & in the (a > b) & (b .> c) that comes from a > b .> c. Or we could do something extreme like push the decision on whether > is being used as a comparison to a later stage, with possible performance costs.

[carlobaldassi]

Thanks. The original design is indeed very nice and easy to extend (once you get it...).
The precedence issue doesn't trouble me too much after all. But one more thing I wanted to add was redirection operators (like 2>&1 in the shell, so that you can pass along everything to a pipe for example) and in that case the syntax would get even farther from standard shell. So maybe having a more general redirection operator would do it, e.g. passing it a Dict of redirections:

`cmd` >& {1=>"out.txt", stderr_stream=>"err.txt"} # two redirs at a time
`cmd` >& {2=>1} | `cmd2` # pipe everything to cmd2
`cmd` >& {3=>1,1=>2,2=>3} # flip out and err

...uhm, time for me to go to sleep I guess.

[nolta]

So maybe having a more general redirection operator would do it, e.g. passing it a Dict of redirections: ...

Reminds me of the plan9 shell syntax: http://plan9.bell-labs.com/sys/doc/rc.html

[StefanKarpinski]

One could try to exploit the call to & in the (a > b) & (b .> c) that comes from a > b .> c.

That seems like a pretty reasonable idea although I'm not certain how to make it work. I guess that b .> c would have to return something that basically ignores the b and is a standalone redirection object which, when combined with a Cmd object with & does the desired redirection. I also like the hash idea — it's a very nice, clean, programmatic syntax. They're not mutually exclusive, of course. I would say that the last example should just be written as:

`cmd` >& {1=>2,2=>1}

We can figure out that a temporary is needed to implement that. It's a little harder to program, but much easier to use.

[carlobaldassi]

Reminds me of the plan9 shell syntax: http://plan9.bell-labs.com/sys/doc/rc.html

wow, it really does. I had no idea about that. BTW while reading I recalled that I had a question: is there a way to specify /dev/null in an OS-independent way within Julia? If not, I think it would be handy.

Anyway, the Dict idea has at least one drawback: as proposed, one can't specify modes (e.g. if one wants to append to stdout).

First solution: have different operators >&, >>&. The first one would accept everything, the second one only redirections to files specified as Strings. Other operators like |& or >>>& may have some merit, but I'm not sure about it.

Second solution: optional annotations, e.g. something like >& {1=>("out.txt", "a"), 2=>"err.txt"}.

I don't have a preference.

I guess that b .> c would have to return something that basically ignores the b and is a standalone redirection object which, when combined with a Cmd object with & does the desired redirection.

The problem here is that both b and c can be Strings, for which > and .> are already defined.

BTW another slightly annoying precedence issue is that (cmd.> "err.txt") |cmd2`` also requires brackets, or | will take precedence.

I guess that there are just a few ways to deal with all this: 1) give up and keep the brackets 2) change notation and abandon comparison operators (e.g. use >&, .>& and friends as redirection operators, determining what to do depending on the type of the rhs) 3) use a macro and pre-parse everything in order to rearrange the arguments, then pass it over to the functional form.
I can't think of anything else.
Number 3 is probably the worst.

[pao]

(3) isn't so bad if it's part of the backtick parser. cmd > err.txt | cmd2?

[carlobaldassi]

Sorry, I just realized that the way I had implemented here-strings had a buffering problem, and besides it was just silly. I fixed it, and it's also much cleaner now.

[StefanKarpinski]

(3) isn't so bad if it's part of the backtick parser. cmd > err.txt | cmd2?

I've actually considered this because it would be nice to be able to write readall(cat foo.txt | sort | uniq) and have it work. At that point, we're basically doing a Julia shell implementation, which is ok by me. The thing that gets a little weird is what to do with foo` & `bar, which at the Julia level means "do these things in parallel, independently", but does not mean that in any shell. I kind of like having the backticks be commands only and having all operations occur in real code. Carlo, you've been working with this stuff a lot now, any thoughts?

[carlobaldassi]

Carlo, you've been working with this stuff a lot now, any thoughts?

Disclaimer: I've only been working on the internals, I've never even read the backticks parser's code. That said, I think building a whole Julia shell within backticks will lose a lot in term of programming capabilities. Maybe if it's just redirections (possibly including pipes, which however would need to also keep their current syntax) it could make sense, as long as it doesn't bring in the need for the escaping-rules hell which shells (the ones I know of at least) are notorious for; I'm not sure that's possible, but I think I could try to prototype something in a few days if I find the time and if it's ok with you.

[StefanKarpinski]

Let's try to come up with something that's not at the backticks syntax level. I'm ok with having >&, .>&, >>& and .>>& be new "redirection operators", and I'm also ok with the hash syntax idea. Let's play with those a bit more before we resort to putting things into the backtick syntax itself.

[StefanKarpinski]

BTW while reading I recalled that I had a question: is there a way to specify /dev/null in an OS-independent way within Julia? If not, I think it would be handy.

This could be represented with redirection to nothing, but I'm not sure about that. Is there an equivalent to /dev/null on Windows?

[pao]

For Windows you want NUL.

[carlobaldassi]

Let's play with those a bit more before we resort to putting things into the backtick syntax itself.

Fine by me!

Is there an equivalent to /dev/null on Windows?

In the DOS prompt it used to be NUL, but that's not a file. Wikipedia also speaks about \Device\Null, but I don't understand the following explanation about it being inaccessible. In this discussion it's suggested that nul would be used in C code. Unsurprisingly, it seems to be a mess.

[Keno]

NUL works always.

[StefanKarpinski]

Shall we merge this then? It looks good to me. A lot of stuff will have to be rewritten when we merge the Windows port, but that's ok.

[StefanKarpinski]

I'm going to merge. We can always revert if anyone starts yelling about it ;-)