Skip to content

Provide for ETL Chain Splitting  #35

@justb4

Description

@justb4

There are cases where we would like to split a Stetl ETL Chain: for example to publish converted data to both a database and a CSV file or some or multiple web API(s). Most ETL frameworks provide a Splitter (and a Combiner or Merger, which is also handy but trickier and less required).

Implementation considerations

This could be built into to internal Stetl base classes. This would require a change in the .ini-file notation, examples:

Simplest case is Output splitting using the () notation as we may want to split into sub-Chains:

input | filter | (output1) (output2)

Splitting into sub-Chains at Filter-level:

input1 |filter1 | (filter2a | output1a) (filter2b | output1b)

to split the output of filter1 into sub-Chains filter2a | output1a and filter2b | output1b.

Combining could use similar notation:

(input1)(input2) | filter | output

Or even splitting + combining:

(input1)(input2) | filter | (filter2a | output1a) (filter2b | output1b)

Another option is a specialized Filter or Output. In the latter case like a CompositeOutput which is parameterized with a list of Outputs that it calls upon. Disadvantage is that the Chain configuration is hidden in the Composite Component's config.

In first instance, simple Chain splitting will be provided with this issue. A Combining implementation will be done in a separate issue.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions