If you want to use nautirust but not develop for it, take a look at nautirust-configs.
Nautirust is a command-line program that helps configuring, and is able to start, a data processing pipeline of processes (called steps). It does not restrict on programming language or start-up sequence.
Nautirust consists of 3 concepts: 1. a step, 2. a runner and 3. a channel. A pipeline is a sequence of steps.
Runners are designed to start a step and can require steps to define a configuration. For example a runner that executes a step written in javascript requires the location of the main source file and the name of the function that will be executed.
Each step is programming language independent, but has to specify what runner must be used to start the step. A step also specifies what arguments should be provided to that step, including the type.
The type of these arguments is loose but there are two special cases: stream reader and stream writer. Readers and writers connect two processes together with a channel. Nautirust helps connecting these channels.
The goal of the runner is not only to start up a step, but also to abstract away the underlying channel. This eases the implementation of steps. Runners specify what channels they support. Not all runners can support all channels, nautirust uses this information to only suggest plausible channels.
Example step file (for more details see later)
{
"id": "helloWorld",
"runnerId": "JsRunner",
"config": {
"jsFile": "main.js",
"methodName": "sayHello"
},
"args": [
{
"id": "to",
"type": "string"
}
]
}Generate a plan and save it to plan.json:
nautirust generate -o plan.json [...steps]
Execute the plan:
nautirust run plan.json
Channels and runners have to be defined, this can be done with command line arguments or a config file (orchestrator.toml or specify with the --config flag).
channels = "configs/channels/*.json"
runners = "configs/runners/*/runner.json"This configuration file will look for any channel defined inside configs/channels and will
look for runners defined in configs/runners that have a runner.json file.
Example channel configuration:
{
"id": "file",
"requiredFields": [
"path",
"onReplace"
],
"options": [
{"path": "test1.json", "onReplace": true},
{"path": "test2.json", "onReplace": false},
{"path": "test3.json"}
]
}Here a channel is defined called file and specifies two fields are required: path and onReplace.
It also defines some options for the nautirust. The user can choose between these options.
Example runner configuration:
{
"id": "JsRunner",
"runnerScript": "node ./lib/index.js {config} {cwd}",
"canUseChannel": [
"file", "ws"
],
"canUseSerialization": [
"json", "turtle", "plain"
],
"requiredFields": [
"jsFile",
"methodName"
]
}Here a runner called JsRunner is defined. Required fields are
runnerScript: how is the runner startedcanUseChannel: what channels can this runner provide to the processorcanUseSerialization: what serializations are supported
When a runner is configured in a step jsFile and methodName have to be provided.
Example step configuration:
{
"id": "readCsv",
"runnerId": "JsRunner",
"config": {
"jsFile": "main.js",
"methodName": "writeTo"
},
"args": [
{
"id": "config",
"type": "any",
"description": "config argument"
},
{
"id": "a_default_value",
"type": "string",
"default": true,
"value": "test"
},
{
"id": "a_suggested_value",
"type": "string",
"default": false,
"value": "test2"
},
{
"id": "writer",
"type": "streamWriter",
"targetIds": [
"data"
]
}
]
}A step is a processor. The processor specifies that it can be executed with the JsRunner runner.
It also specifies what arguments have to be defined before being able to execute.
Default arguments are also possible, see a_default_value. However if default is false and a value is still given, it will be suggested to the user instead.
$ nautirust -h
nautirust 0.1.1
USAGE:
nautirust [OPTIONS] <SUBCOMMAND>
OPTIONS:
-c, --channels <CHANNELS> Glob to indicate channels locations
--config <CONFIG> Location of a config file
-h, --help Print help information
-r, --runners <RUNNERS> Glob to indicate runners locations
-V, --version Print version information
SUBCOMMANDS:
docker Create a docker-compose file from a nautirust pipeline
generate Generate a pipeline of steps
help Print this message or the help of the given subcommand(s)
prepare Prepares the execution pipeline by starting the required channels/runner
run Run a configured pipeline
stop Gracefully stop the runners and channels specified in the config
validate Validate configureations for runners and channelsYou can specify (by glob) where to find channels and runners with --channels and --runners respectively.
Nautirust also supports toml file that specifies these properties (--config).
$ nautirust generate -h
nautirust-generate
Generate a pipeline of steps
USAGE:
nautirust generate [OPTIONS] [STEPS]...
ARGS:
<STEPS>... Steps to include in the pipeline (ordered)
OPTIONS:
-a, --automatic Try infer basic configurations details
-h, --help Print help information
-o, --output <OUTPUT> Output location of the generated pipeline fileNautirust takes multiple steps to create a pipeline configuration file. Interactively the user is asked questions about the required arguments, taking special care of stream readers and stream writers.
These questions can add up, use the -a flag to let nautirust infer some basic configuration consisting of:
- automatic linking of stream readers and writers with the same name
- automatically choosing a channel configuration when the channel type is specified
-o takes a filename to store the generated configuration (default is stdout).
$ nautirust run -h
nautirust-run
Run a configured pipeline
USAGE:
nautirust run [OPTIONS] <FILE>
ARGS:
<FILE> Config file
OPTIONS:
-h, --help Print help information
-t, --tmp-dir <TMP_DIR> temporary directory to put step configuration filesNautirust runs a generated configuration file.
Each runners takes in a configuration file that specifies the steps that should be executed, with -t you can specify the location of these configuration files.
$ nautirust prepare -h
nautirust-prepare
Prepares the execution pipeline by starting the required channels/runner
USAGE:
nautirust prepare <FILE>
ARGS:
<FILE> Config file
OPTIONS:
-h, --help Print help informationNautirust takes a generated configuration file, and prepares the used steps, runners and channels. This can be used to run a build script, start a docker-compose instance, ...
$ nautirust stop -h
nautirust-stop
Gracefully stop the runners and channels specified in the config
USAGE:
nautirust stop <FILE>
ARGS:
<FILE> Config file
OPTIONS:
-h, --help Print help informationSame as prepare, but in reverse.
EXPERIMENTAL
$ nautirust docker -h
nautirust-docker
Create a docker-compose file from a nautirust pipeline
USAGE:
nautirust docker [OPTIONS] <FILE>
ARGS:
<FILE> Config file
OPTIONS:
-h, --help Print help information
-o, --output
-t, --tmp-dir <TMP_DIR> temporary directory to put step configuration filesDoes the same as nautirust run but generates a docker-compose file that when executed start the pipeline.
This only works when the runners have specified a docker command.
This docker command is expected to generate a Dockerfile somewhere that executes the step.
This command also prints to stdout the required contents for this step (at least specify the generated Dockerfile and set up the correct docker context).
The used channels also return a part of the docker-compose file (if anything).
$ nautirust validate -h
nautirust-validate
Validate configureations for runners and channels
USAGE:
nautirust validate
OPTIONS:
-h, --help Print help informationValidates the specified channels and runners.