Support split config across the pipeline platform

[zhitaoli]

We need to support specifying a list of splits in various example_gen implementations and corresponding distribution, and make sure all components can select which splits they process.

[aaronlelevier]

Not sure if here or #20 is better to start the conversation.

From reading the code, it seems like the split information would be needed by CsvExampleGen.checkcache because the data may exist but if a new split is being specified, then you'd want to cache bust and run CsvExampleGen.exec to avoid re-using an old split with the wrong split percentage

[zhitaoli]

There are several levels for custom config:

1. Supporting a different train/eval ratio (which is How to set the size of train-eval split with CsvExampleGen? #20 talking about): I think this should be mostly local to example_gen classes and should be easier to implement;
2. Supporting different split names from hard-coded train/eval: this is what Support split config across the pipeline platform #21 is talking about and would affect all components dealing with examples.

We hope to publish a RFC/design doc soon about 2), which will require 1).

[zhitaoli]

Some update: step 1 (supporting a different train/eval ratio) is done in #20. We don't have an estimate on next step yet, but please let us know if you are blocked by it and we will revisit.

[1025KB]

Custom split is supported, check this guide