A framework for creating and running data pipelines. Data pipelines have stages, which are made of an arbitrary number of tasks. Stages and tasks are run in serial: once a task completes, the next task in the pipeline is executed.
A task may complete synchronously, or asynchronously through use of a promise.
Pipelines take a data object as input, and each task operates on the object in some way.
For an example of the Pipeliner in action, checkout the nxus-static-site module.
> npm install nxus-pipeliner --save
import {pipeliner} from 'nxus-pipeline'
pipeliner.pipeline('my-pipeline')
A task is a javascript function that accepts any objects passed into the pipeline when it is run. Tasks are run serially in FIFO order.
let myTask = (word) => {
word.toUpperCase();
}
app.get('pipeliner').task('my-pipeline', myTask)
Once all the tasks for a pipeline have been defined, the last step is to run the pipeline.
app.get('pipeliner').run('my-pipeline', someData)
Extends NxusModule
Parameters
app
Examples
let myTask = (data) => {
data.word.toUpperCase()
}
let data = {word: 'hello'}
let pipeliner = app.get('pipeliner')
pipeliner.pipeline('capitalize')
pipeliner.run('capitalize', data).then(() => {
console.log('data') // {word: 'HELLO'}
})
Create a new pipeline configured with three stages: 'collect', 'process', and 'generate'. Does nothing if the named pipeline already exists.
Parameters
pipeline
string The name of the pipeline to create
Defines a task for a pipeline and a stage. Creates the pipeline if it does not already exist; adds the stage if it does not already exist. If multiple tasks are defined for a stage, they are run in the order defined.
Parameters
Returns all pipelines which have been defined.
Returns object A hash of the pipelines.
Returns a specific pipeline.
Parameters
pipeline
string The name of a pipeline to return.
Returns object The pipeline object.
Runs the specified pipeline, passing the arguments to each task.
Parameters
pipeline
string The name of the pipeline to runargs
...object Arguments to pass to the pipeline tasks
Returns Promise A promise that is fulfilled when the pipeline completes; it is rejected if any task in the pipeline fails (throws an error or returns a promise that is rejected)