Design for scsh/shell-like process control

[pclouds]

I'm implementing "||", "&&" and "|" shell syntax in scheme. Maybe it could become part of gauche.process. Before I waste your time with my code. I'd like to check what would be a good design for this. What I have in mind is this. A bit higher level than run-process, but still not the same as scsh. But hopefully we can make scsh wrapper more easily with these.

Process Form

A process is represented by

• A single string

• A single symbol

• A list of strings or symbols (could be intermixed)

• A list, where the first item is in either three above forms, followed by a list of run-process-like redirects, e.g.

  ((ls) (> 1 "out") (< 0 "in"))

Process execution

The procedure "run-pf&" takes one process form (pf), executes it and return the process object immediately. This is simply a wrapper around run-process that can handle process forms.

The procedure "run-pf" is like run-pf& except that it waits for the process to terminate and returns the exit code. If the process is signaled, the exit code is in 128+ range like how unix shell does it. Anything that can't be converted to exit code (e.g. process-wait fails) results in an exception.

run-pf can take more than argument too. In that case it executes processes sequentially like scheme "begin" form and returns exit code of the last process.

The procedure "run-and" takes 1+ arguments as process form and executes them in sequence so long as their exit codes are zero, i.e. "&&" syntax. It returns exit code of the last executed process. "run-or" is similar, for "||" syntax.

Redirection is generally ok for run-pf, run-and and run-or, except redirection to ports, for obvious reasons.

Piping

The procedure run-pipe+& has the following form

(define (run-pipe+& first-redir last-redir connect-list . pf/rest))

first-redir and last-redir are two lists of extra redirections for the first and the last process. They are in the same form that of :redirects keyword from run-process. This allows us to setup stdin of the first process, or stdout of the last process.

connect-list specifies how to connect two consecutive processes together. It's the exact same form from scsh's fork/pipe+. For example "((1 2 0) (3 3))" says redirect both stdout and stderr of the first process to stdin of the second one, and redirect fd 3 of the first to 3 of the second.

run-pipe+& immediately returns a list of process objects. For example, we could offload complex processing to some oustide command with

(run-pipe+& '((< 0 input)) '((> 1 output)) '((1 0)) '(cmd1) '(cmd2) '(cmd3))

Then retrieve the two input/output ports from the relevant process objects and start piping.

From that we have a few convenient wrappers:

• run-pipe+ waits for all processes to finish before return, and returns exit code of the last process in pipe. It does not take first-redir/last-redir either
• run-pipe is run-pipe+ with default connection-list ((1 0))

[shirok]

scsh-type shell control has long been on my todo list. One problem I see in scsh approach is that it introduces sort of mini-language but with confusing syntax (i.e. comnand arg list and redirections are both represented by lists. For example, (ls -l) and ((ls -l) (> 1 "abc")) both represent a process, but in the first one the outer list represents command and its args, while the second one the outer list have different meaning.

I feel we need either (1) an DSL a lot more tuned for shell programming, or (2) more compositional approach, e.g. each process form becomes a first-class object and we combine them much like function composition.

I vaguely remember there was a discussion among uses several years back. I'll dig it up if I can find it.

[pclouds]

Thanks. I will also look at Haskell and see how they deal with this. A quick search shows that they do have shell-like functions.

[xificurC]

Looking at some examples I don't see how yours would work @shirok . From what I can see each process in scsh notation is a non-nested list, although I admit I haven't checked all the procedures. When trying your example in chicken-scheme's implementation I get an error. The correct syntax would be (run (ls -l)) in the first case which was correct and (run (ls -l) (> "abc")) in the second.

I understand the opinions can vary on the process notation though. Personally I found it takes a while to get it right but is composable and to the point. Either way getting a mini-DSL or a more powerful API to handle processes would be greatly welcomed by me as I could drop writing bash scripts to glue external commands together. The chicken implementation of scsh-process hits in 400 loc and could probably be easily ported to gauche, but I'm not going to as far as looking into it if you have a different solution in mind :)

[pclouds]

Just a bit of update from my side if anyone is wondering. I'm so busy with stuff that I still have not investigated any further :(

[shirok]

@xificurC, in the development HEAD I've added do-process and do-process-pipeline in gauche.process. These wrap the counterparts of run-* version for the typical case that you run process synchronously and only care about its exit status. With them, typical process operations (shell's && and || and conditionals) can be written as mere Scheme expressions. Document is half-done and sitting in my local branch, though.

The advantage is that users don't need to remember yet another DSL to handle all possible options and various ways to combine processes. The disadvantage is it's more verbose than process forms. Some of verbosity can be addressed by tuning the API, I think.

[xificurC]

@shirok I remember you adding those, it actually stemmed from a mail question I had, thanks :)

Maybe if there was an API with a bit higher-level coverage that would allow all of redirection and pipelining options it would suffice. If not it would be a good base for someone to wrap that in another scsh (or something alike) DSL for his/her own needs. I understand you're not keen on building DSLs out of everything, but when one wants to use something extensively it really pays off.

[pclouds]

There is 7424fd7 (Introduce do-pipeline/run-pipeline - 2017-01-10). Together with do-process, process-output->... I think we have covered pretty much all common cases (the other two are maybe string->process-input and something that does (cat port) | command | (port->string-list) or something along that line, where the port is not a file.

Anyway I think I'm closing this ticket since it's good enough, and I did not make any progress at all about the DSL front. It'll come up when it comes up.

[shirok]

Thanks. For the uncovered cases, feel free to open issues if you can identify a specific common pattern you want to support.