Running External Commands

Running External Commands

WORK IN PROGRESS, EARLY STAGE

This page provides an overview of syntax and other facilities available in Next Generation Shell for running external programs.

Syntax

$(my_prog ...)

Runs external program (or pipeline)

Example:

if $(test -f my_file) {
	...
} else {
	...
}

• Captures stdout and stderr (typically)
• Waits for the program to finish (unless & is used, see below)
• Returns ProcessesPipeline
• Throws ProcessFail on error. See ok:VALUE and ok_sig:VALUE value below for how to control which exit codes and signals cause this exception to be thrown.

See also: "Inside $(), $[], %(), single-backtick, double-backtick"

Single-backtick

Runs external program and captures the output as single string (Str type in NGS).

In contrast to other shells, NGS does not strip the last newline. While it's practical in many cases to strip, it also loses information: you are unable to distinguish whether the newline was or wasn't there.

Double-backtick

Runs external program and parses the output.

Examples:

my_data_structure = ``curl https://example.com/api/list-of-objects.json``

instances = ``aws ec2 describe-instances``

Special treatment for AWS CLI output

To provide better ergonomics, several transformations are made to output of AWS CLI command.

In the instances example below, instances will be array (Arr) of instances so no Reservations -> Instances traversal is needed.

Many AWS CLI commands return a Hash where only one meaningful key has an Arr as value. That array is returned and no .MyOnlyMeaningfulField is needed.

Customizing parsing for your own commands

TODO: Desribe extending through decode().

$[my_prog ...]

Same as $(...) but without capturing standard output, letting it flow to the standard output of the parent process (the NGS script).

Example:

echo("Your files:")
$[ls -l]

%(my_prog ...)

A literal for external program or pipeline. Like $(...) but without running.

%[argN ...]

Array of literals.

This is not a syntax that is directly involved in running external programs but rather used frequently in conjuction to construct arguments.

Example:

files = %[file1.txt file2.txt]
$(touch $*{files})

Same as:

files = ['file1.txt', 'file2.txt']
$(touch $*{files})

Inside $(), $[], %(), single-backtick, double-backtick

This section describes syntax available

$(HERE)
$[HERE]
%(HERE)
`HERE`
``HERE``

${expansion}

$*{expansion}

log:

Logs the command to stderr before executing it.

$ ngs -e '$(log: ls)'
[LOG ...] Running command: ls

ok:VALUE

Overrides which exit codes do not cause ProcessFail exception. For unknown commands, only exit code 0 is ok by default. Known programs hava a list of non-error exit codes.

Example:

$(ok:1 diff old.txt new.txt)  # exit 1 means "files differ", not an error

VALUE can be:

• Int — exact match (e.g. ok:1)
• NumRange — in range (e.g. ok:0..10)
• Arr — in list (e.g. ok:[0,1,2])
• bare ok: — any exit code is ok

Note that unlike in many other shells, failure and boolean false are not conflated. The example above shows how non-zero exit status does not cause ProcessFail exception but still evaluates to false when converted to Bool (done by if). Typical bash script, conflates the two concepts and the typical if test -f my_file would take the else branch for both missing file and any other error, for example syntax error of test arguments: if test --f my_file and you are lucky if you notice the stderr showing the error.

NGS: if $(...) has 3 branches - true, false, and exception.

Bash: if ... has two branches, where the false branch is taken for both boolean false and any other arbirary error.

ok_sig:

Termination by the given signal is not an error (e.g. ok_sig:SIGNALS.TERM).

top_level::

TODO: old but useful when composing, give example of modifying a command - a command that we decided should always pass the ouput through (maybe for debugging)

line:

&

cd:

env:

<file

N>file

TODO: N is optional, redirect file descritor to the given file, overwriting the file

N>>file

TODO: N is optional, redirect file descritor to the given file, appending to the file

2>${true}

TODO: Capture stderr

>${false}

TODO: Do not capture stdout

|

Types

ProcessesPipeline

Process

Command

CommandsPipeline

CommandRedir

Program

TODO: Example of Program(...) or Program(...), finch and docker.

ProcessRedir

Pipe

ProcessFail

ProgramNotFound

PipeCreateFail

KillFail

Methods

finished_ok()

wait()

kill()

lines()

Str()

Bool()

TODO: Bool of all relevant types: Program, Process, etc

TODO: Relation to assert()

finished()

exec()

... and friends

write()

echo()

read()

close()

assert(p:Program)

Arg()

TODO. Compared to Str, this is specifically to be used as external program argument.

Argv()

Facility for constructing arguments for external programs.

Environment

TODO: ENV, env:, give examples of {} + ENV, ENV + {}, ENV.dflt(...)