processchain wait/wait_success issues

[swadey]

I'd like to run a process chain and wait for it to finish. Unfortunately it looks like:

1. wait is not defined for process chains, and
2. wait_success uses success() which defines success like this:

proc.exit_code==0 && (proc.term_signal == 0 || proc.term_signal == SIGPIPE)

Unfortunately, grep defines success as 0 or 1 (sigh):

EXIT STATUS
     The grep utility exits with one of the following values:

     0     One or more lines were selected.
     1     No lines were selected.
     >1    An error occurred.

Here's an example:

julia> x = readsfrom(`ls` |> `grep xxxxxx`)
(NamedPipe(closed, 0 bytes waiting),ProcessChain([Process(`ls`, ProcessExited(0)),Process(`grep xxxxxx`, ProcessExited(1))],SpawnNullStream(),NamedPipe(closed, 0 bytes waiting),TTY(open, 0 bytes waiting)))

julia> wait(x[2])
ERROR: no method wait(ProcessChain,)

julia> Base.wait_success(x[2])
false

[vtjnash]

Since grep defined success as 0, I don't understand the bug. wait_success also uses wait_exit, which should not check any status codes, and success should be respecting the ignorestatus function call for a cmd. wait is not defined as it is ambiguous. Can you clarify your issue?

[JeffBezanson]

wait is defined for Process, so it could be for ProcessChain too. I suppose it should mean waiting for the last process in the chain?

[vtjnash]

That logic flows equally well in the opposite direction and should not be the basis for an argument. I suppose that wait is not defined for ProcessChain so it should not be defined for Process either.

I had remove the concept of a generic wait for processes and process chains (for the libuv fork), because it was a source of confusion and a few bugs in the handling of processes a year ago due to the ambiguity. I was not aware that it had been redefined, and recommend it be removed (e.g. revert d94d15f).

[JeffBezanson]

I said could, not should.

Perhaps the most fundamental operation on a child process is to wait for it to exit and get the exit code, so we need to have that. What is the source of the ambiguity?

Should we instead use wait_exit? I guess I'm ok with that if it's updated to return the exit code.

[vtjnash]

For example: Is it waiting on the first, last, or all processes? Are you waiting for them to be started or to exit? Are you waiting for them to be able to accept data, or to be able to read data from the output? all data, or any data?

Yes, wait_exit could return the exit code, similar to how wait_success returns the success code. I think we might be able to do something of the form wait(:exit, ...), but that would be a API redesign and not in the immediate scope of 0.2

[swadey]

Grep defines success as 0 or 1. See my clip of the man page for grep.

0 - found something
1 - found nothing
2 or greater - error

thanks,
wade

On Jul 26, 2013, at 11:57 PM, Jameson Nash notifications@github.com wrote:

Since grep defined success as 0, I don't understand the bug. wait_success also uses wait_exit, which should not check any status codes, and success should be respecting the ignorestatus function call for a cmd. wait is not defined as it is ambiguous. Can you clarify your issue?

—
Reply to this email directly or view it on GitHub.

[swadey]

I'm waiting for exit of the whole processchain (in this case a pipeline).

Btw wait_success is not exported explicitly. It doesn't look like it is supposed to be called directly?

thanks,
wade

On Jul 27, 2013, at 1:56 AM, Jameson Nash notifications@github.com wrote:

For example: Is it waiting on the first, last, or all processes? Are you waiting for them to be started or to exit? Are you waiting for them to be able to accept data, or to be able to read data from the output? all data, or any data?

Yes, wait_exit could return the exit code, similar to how wait_success returns the success code. I think we might be able to do something of the form wait(:exit, ...), but that would be a API redesign and not in the immediate scope of 0.2

—
Reply to this email directly or view it on GitHub.

[swadey]

Sorry, I should have been more clear. There are two problems:

1. Grep returns non-error conditions for return code 0 or 1:

EXIT STATUS
     The grep utility exits with one of the following values:

     0     One or more lines were selected.
     1     No lines were selected.
     >1    An error occurred.

Note that status >1 indicates error. 1 indicates that grep has empty results. Since this is program specific, is there a way to specificy a custom condition on the Process object to interpret it's return code?

1. there doesn't seem to be a public API way to wait for processchain (in this case a pipeline) to finish. I want to run mulitple asynchronous pipelines then wait for them to complete then check for errors. wait_success is the closest, it's not public. wait() was the first place I thought to look given that you can wait on individual processes.

[JeffBezanson]

@vtjnash I'm not totally convinced --- historically, wait/waitpid wait for processes to exit. Waiting for a process to start is not really a thing. If you're waiting for data, you'd be using one of the process' i/o streams and not the process itself. It also seems clear, at a high level, that waiting for a process chain means waiting for the whole thing (which in practice means waiting for the last process). Waiting for a process in the middle of a chain would be a fairly exotic case.

[JeffBezanson]

Now wait can wait on a process or chain, and the exitcode fields of processes should be examined to see what happened. success also waits, but provides a default behavior of (1) assuming an exit code of 0 means success, (2) raising an exception if the process could not start.