Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Adding Multiple Training Files to the Pipeline? #192
In the Taxi Fare example, just adding another textloader and/or ColumnCopier, etc seems to not be correct.
changed the title from
.Net Framework Support?
Adding Multiple Training Files to the Pipeline?
May 19, 2018
Thanks for asking! This is not currently possible, but let's use this issue to track enabling multiple inputs in a pipeline.
Just to clarify: is your intention to concatenate the two files as soon as they are loaded, or to apply different transforms/trainers to them?
A potential workaround for now is to read in the examples from both files into memory and use the
My intention is for creating and testing ML structures with large datasets to be modular and less taxing on file transfers to/from servers. For example, moving 100GB is to a server is easier if split by time or another parameter. It also allows ML structures to be updated as new data comes in without having to concat onto what are already are/is a large file.
Reducing the memory footprint by loading subsets of the data would be nice, but as I understand it, that is not possible for all ML structures.
I have concated the files and it works properly but this would be a nice feature to have.
Thanks for the answer.