Pipelines

Pipelines

A Shelly pipeline connects stages with |. Between Shelly stages, the item is a value with its type intact. Between external programs, the connection is a Unix byte stream.

Transform each item

A source collection supplies items through the next_item protocol. A function with no explicit arguments receives the current item as its one argument:

fn square($n: Integer): Integer { $n * $n }
let $squares: [Integer]
[3, 6, 9] | square | $squares
echo $squares...

9 36 81

Bare $ means the incoming item. Named variables retain their ordinary meaning. Once a function stage has explicit arguments, include $ to say where the item goes. Parentheses make an expression into a stage.

fn badge($prefix: String, $number: Integer): String { "${prefix}-${number}" }
let $badges: [String]
[4, 7] | ($ * 10) | badge "desk" $ | $badges
echo $badges...

desk-40 desk-70

Only the source is automatically iterated. An array returned by a stage remains one array item; a later stage can spread it into function arguments with $.... A stage returning () passes that value onward. It does not forward an earlier item or silently filter the item out.

To send an upstream stream down several paths, use visual pipeline splits. Each branch is connected to a specific pipe by aligned guides.

Collect at the end

A final bare variable is an assignment destination. Declare it first. An array destination collects the results; a scalar destination retains the last result. An empty source clears an array destination and leaves a scalar destination alone.

fn total($numbers: Number...): Number { $numbers.sum }
let $totals: [Number]
[[2, 3], [8, 1], [4, 4]] | total $... | $totals
echo $totals...

5 9 8

Programs, data, and status

producer | consumer preserves ordinary streaming bytes when both stages are external programs. Use format annotations or the conversion protocol when structured data crosses a program boundary. An unannotated external source becomes text when entering a Shelly stage.

A pipeline containing external programs returns an aggregate ExecResult status. Its data travels through stages or into the final destination. Capturing the status lets the caller handle failure. Without capture, an unhandled failure in any external stage fails the pipeline. A pipeline made entirely of Shelly values and functions returns its final data value.

let $text: String
let $status = printf "north\nsouth\n" | head -n 1 | $text
echo $text.trim $status

north ExecResult(0)

Pipelines associate left to right. Parenthesize a whole pipeline when using its result inside another expression.

---

Related: Pipeline formats · Iteration · Exit status