Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide for ETL Chain Splitting #35

Closed
justb4 opened this issue Jun 8, 2016 · 1 comment
Closed

Provide for ETL Chain Splitting #35

justb4 opened this issue Jun 8, 2016 · 1 comment
Assignees
Labels
Milestone

Comments

@justb4
Copy link
Member

@justb4 justb4 commented Jun 8, 2016

There are cases where we would like to split a Stetl ETL Chain: for example to publish converted data to both a database and a CSV file or some or multiple web API(s). Most ETL frameworks provide a Splitter (and a Combiner or Merger, which is also handy but trickier and less required).

Implementation considerations

This could be built into to internal Stetl base classes. This would require a change in the .ini-file notation, examples:

Simplest case is Output splitting using the () notation as we may want to split into sub-Chains:

input | filter | (output1) (output2)

Splitting into sub-Chains at Filter-level:

input1 |filter1 | (filter2a | output1a) (filter2b | output1b)

to split the output of filter1 into sub-Chains filter2a | output1a and filter2b | output1b.

Combining could use similar notation:

(input1)(input2) | filter | output

Or even splitting + combining:

(input1)(input2) | filter | (filter2a | output1a) (filter2b | output1b)

Another option is a specialized Filter or Output. In the latter case like a CompositeOutput which is parameterized with a list of Outputs that it calls upon. Disadvantage is that the Chain configuration is hidden in the Composite Component's config.

In first instance, simple Chain splitting will be provided with this issue. A Combining implementation will be done in a separate issue.

@justb4 justb4 added the enhancement label Jun 8, 2016
@justb4 justb4 added this to the Version 1.0.9 milestone Jun 8, 2016
@justb4 justb4 self-assigned this Jun 8, 2016
@justb4 justb4 modified the milestones: Version 1.10, Version 1.0.9 Jun 17, 2016
@justb4 justb4 changed the title Provide for Splitting an ETL Chain Provide for Output Splitting an ETL Chain Dec 13, 2016
justb4 added a commit that referenced this issue Dec 13, 2016
@justb4 justb4 changed the title Provide for Output Splitting an ETL Chain Provide for ETL Chain Splitting Dec 14, 2016
justb4 added a commit that referenced this issue Dec 14, 2016
@justb4
Copy link
Member Author

@justb4 justb4 commented Oct 20, 2017

Splitting now ready, Merging still todo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
You can’t perform that action at this time.