You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
There are cases where we would like to split a Stetl ETL Chain: for example to publish converted data to both a database and a CSV file or some or multiple web API(s). Most ETL frameworks provide a Splitter (and a Combiner or Merger, which is also handy but trickier and less required).
Implementation considerations
This could be built into to internal Stetl base classes. This would require a change in the .ini-file notation, examples:
Simplest case is Output splitting using the () notation as we may want to split into sub-Chains:
Another option is a specialized Filter or Output. In the latter case like a CompositeOutput which is parameterized with a list of Outputs that it calls upon. Disadvantage is that the Chain configuration is hidden in the Composite Component's config.
In first instance, simple Chain splitting will be provided with this issue. A Combining implementation will be done in a separate issue.
The text was updated successfully, but these errors were encountered:
There are cases where we would like to split a Stetl ETL Chain: for example to publish converted data to both a database and a CSV file or some or multiple web API(s). Most ETL frameworks provide a Splitter (and a Combiner or Merger, which is also handy but trickier and less required).
Implementation considerations
This could be built into to internal Stetl base classes. This would require a change in the
.ini
-file notation, examples:Simplest case is
Output
splitting using the () notation as we may want to split into sub-Chains:Splitting into sub-Chains at
Filter
-level:to split the output of
filter1
into sub-Chainsfilter2a | output1a
andfilter2b | output1b
.Combining could use similar notation:
Or even splitting + combining:
Another option is a specialized
Filter
orOutput
. In the latter case like aCompositeOutput
which is parameterized with a list ofOutput
s that it calls upon. Disadvantage is that the Chain configuration is hidden in theComposite
Component's config.In first instance, simple Chain splitting will be provided with this issue. A Combining implementation will be done in a separate issue.
The text was updated successfully, but these errors were encountered: