Generalize handling of merged TypedPipes #975
Conversation
| * Cascading can't handle duplicate pipes in merges. What we do here is see if any pipe appears | ||
| * multiple times and if it does we can do self merges using flatMap. | ||
| * Finally, if there is actually more than one distinct TypedPipe, we use the cascading | ||
| * merge primitive. |
There was a problem hiding this comment.
while in here maybe just mention that we do a rename all the individual pipes -- or are we just assigning names now?
There was a problem hiding this comment.
right now, we just make an identity pipe (new Pipe(newName, previous)) with a new unique (just integer increment now) name, but only if there is more than one pipe going into the merge.
I'm not totally clear what I should add. Suggestions?
There was a problem hiding this comment.
Just a note that we must rename the pipes if we use the merge call since cascading can't handle the multiple pipes possibly with the same name? -- "When using the merge primitive we rename all pipes going into it as Cascading cannot handle multiple pipes with the same name."
|
LGTM, had a quick question on comment. Not important. Build failure looked a bit spurrious. Restarted. Merge when green with comment change if you think appropriate...fine without too imo. |
Generalize handling of merged TypedPipes
This flattens merges all the way out into a List, then checks for duplicates in that list. If they exist, do a local self merge leaving us with only distinct pipes, which cascading can handle.