-
Notifications
You must be signed in to change notification settings - Fork 33
Description
There are cases where we would like to split a Stetl ETL Chain: for example to publish converted data to both a database and a CSV file or some or multiple web API(s). Most ETL frameworks provide a Splitter (and a Combiner or Merger, which is also handy but trickier and less required).
Implementation considerations
This could be built into to internal Stetl base classes. This would require a change in the .ini-file notation, examples:
Simplest case is Output splitting using the () notation as we may want to split into sub-Chains:
input | filter | (output1) (output2)
Splitting into sub-Chains at Filter-level:
input1 |filter1 | (filter2a | output1a) (filter2b | output1b)
to split the output of filter1 into sub-Chains filter2a | output1a and filter2b | output1b.
Combining could use similar notation:
(input1)(input2) | filter | output
Or even splitting + combining:
(input1)(input2) | filter | (filter2a | output1a) (filter2b | output1b)
Another option is a specialized Filter or Output. In the latter case like a CompositeOutput which is parameterized with a list of Outputs that it calls upon. Disadvantage is that the Chain configuration is hidden in the Composite Component's config.
In first instance, simple Chain splitting will be provided with this issue. A Combining implementation will be done in a separate issue.