RFC: deprecate readsfrom/writesto in favor of simply using open on AbstractCmd

[stevengj]

Supersedes #6945. Now you can simply do e.g.

open(readline, `ls`, "r")

to read from a command or e.g.

open(`less`, "w", STDOUT) do io
    for i = 1:1000
          println(io, i)
    end
end

to write to a command.

The old readsfrom(cmd) is replaced by open(cmd, "r"), and writesfrom(cmd) is replaced by open(cmd, "w").

The deprecated readsfrom and writesto commands are currently used only in 5 packages: Graphs (cc: @pozorvlak, @lindahua), AWS (cc: @amitmurthy), ClusterManagers (cc: @nlhepler), ImageView (cc: @timholy), and WinRPM (cc: @ihnorton, @vtjnash).

cc: @StefanKarpinski, @IainNZ, @loladiro

[stevengj]

(As discussed in #6945, it would be nice to eventually replace readandwrite(cmd) by open(cmd, "r+"), but this seems to require some deeper changes to Pipe.)

[vtjnash]

Does this need close(io) here

[stevengj]

I was thinking that killing P would be enough, but you're probably right that I should close(io) too. However, this may some rearrangement to avoid calling close(io) twice, avoid killing P if we throw from pipeline_error(P), and so on.

[pozorvlak]

Looks good to me, and it would be trivial to port Graphs.jl to the new interface. Display-via-pager would also be a huge win for large graphs!

[JeffBezanson]

Also needs to be removed.

[JeffBezanson]

I like it. What about standard error? Maybe readandwrite should handle that?

[stevengj]

Do you mean, add an extra parameter for stderr?

[StefanKarpinski]

An interesting notion is to bundle all three standard handles into some IOBundle object. Since we tend to write error messages using different function than we print things (i.e. warn, error), this would not be unreasonable. In other words, you could write println(io, "hello") and warn(io, "uh, oh") and the IOBundle object, io, would route the output to the appropriate handle. We're already doing this here for input and output to the process anyway.

[StefanKarpinski]

Of course this is a little different since its the spawned process that is doing the writing to stderr, not us, but having a composite kind of IO object seems relevant anyway.

[stevengj]

It would be nice to have some sh-like pipe operator to redirect STDERR, because you might want to pipe STDERR into some arbitrary process, not just open.

In any case, that should probably wait for a separate PR. Is this PR good to merge?

[stevengj]

It seemed like the changes here were uncontroversial, and the only question is future enhancements like stderr and read/write pipes, so I went ahead and merged.

[BobPortmann]

This

open(`less`, "w", STDOUT) do io
    for i = 1:1000
          println(io, i)
    end
end

sort of works but it does not seem to use less. The output is dumped to the screen w/o paging.

[stevengj]

@BobPortmann, works for me. What OS do you have?

[BobPortmann]

Mac OS X Mavericks 10.9.3

julia> versioninfo()
Julia Version 0.3.0-prerelease+3359
Commit 75f7732 (2014-05-31 18:15 UTC)
Platform Info:
  System: Darwin (x86_64-apple-darwin13.2.0)
  CPU: Intel(R) Core(TM)2 Duo CPU     T9550  @ 2.66GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY)
  LAPACK: libopenblas
  LIBM: libopenlibm

[stevengj]

@BobPortmann, I just tried it on 10.9.3 with 0.3.0-prerelease+3381 (the latest nightly), and writing to less works fine for me.

[BobPortmann]

OK. It works for me too. Not sure what I did different before.

[kmsquire]

The old readsfrom(cmd) is replaced by open(cmd, "r"), and writesfrom(cmd) is replaced by open(cmd, "w").

Would be good to document these as well, especially since they return a (stream, process) tuple. Also should remove docs for old readsfrom and writesto.

[kmsquire]

Nevermind. My helpdb.jl was out of date.

[mhinsch]

Two comments:

• The example in the docs is displayed as text instead of code.
• Is there still a way to get the exit code of the executed process?

[mhinsch]

Also, when using the method open(cmd, mode) there seems to be no way to close the returned stream.

Edit:

Ok, I see now that the return value is a tuple (Pipe, Process), so closing as well as exit code are solved. Still, that's a bit inconsistent with the other uses of open I think.

[JeffBezanson]

Maybe the form of open that accepts a function should return the Process, or a tuple (value, Process).

[StefanKarpinski]

I think returning the process is a good idea. I don't think it's very common to need the value of a open block, so just the process should be fine – you can always use a local variable instead.

[stevengj]

On the contrary, I suspect it's extremely common to want the value of an open block if you are opening it for reading, since x = open(...) do .. end is the easiest way to obtain a scalar value from the read.

I don't see the point of returning the Process in the function form since the Process is no longer running when the open(f::Function, ...) returns.

[mhinsch]

I don't see the point of returning the Process in the function form since the Process is no longer running when the open(f::Function, ...) returns.

I think the idea is to make the exit code of the process available this way (in order to know whether it failed). But I agree, I would imagine the return value of the function to be more interesting than the process/exit code in most cases. Couldn't open instead throw an exception if the process failed?

More generally, however, I find it quite weird to have open return completely different categories of values dependent on how it was called. IMHO this suggests that this is an "overunification" of concepts.

[JeffBezanson]

Ah, I believe open (the block form) does throw an error if the process fails.

[kmsquire]

More generally, however, I find it quite weird to have open return completely different categories of values dependent on how it was called. IMHO this suggests that this is an "overunification" of concepts.

I was thinking the same thing earlier--thanks for stating it in this way, @mhinsch.