Add pipe1.join(pipe2) syntax in TypedAPI #958
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| * This is a class that models the logical portion of the reduce step. | ||
| * in |
|
This also seems safe to me. |
| @@ -660,13 +677,6 @@ final case class MergedTypedPipe[T](left: TypedPipe[T], right: TypedPipe[T]) ext | |||
| MergedTypedPipe(left.hashCogroup(smaller)(joiner), right.hashCogroup(smaller)(joiner)) | |||
| } | |||
|
|
|||
| class TuplePipeJoinEnrichment[K, V](pipe: TypedPipe[(K, V)])(implicit ord: Ordering[K]) { | |||
There was a problem hiding this comment.
With this before was .join not possible?
There was a problem hiding this comment.
You could do this enrichment, but you had to explicitly import it. That is another way to go, just make this class what we enrich to. It more narrowly constrains the enrichment.
That said, it seemed like HashJoinable was the right place to jump in, since we also want pipe1.join(pipe2).join(pipe2) type syntax.
There was a problem hiding this comment.
yep, fair enough. it also moves to the newer .withReducers code paths too.
|
LGTM |
|
Merge when green |
|
Looks great. |
johnynek
added a commit
that referenced
this pull request
Jul 15, 2014
Add pipe1.join(pipe2) syntax in TypedAPI
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
@vitalyg (and others) have argued that doing:
pipe1.group.join(pipe2.group)is quite ugly. This change adds an implicit toHashJoinablewhich is a subclass ofCoGroupable. This means that you can join(K, V)pipes without any extra implicit.This seems safe since the only methods added by the enrichment are the join ones (not the reducing methods on KeyedListLike).
Please speak up if this looks at all dangerous (as implicit conversions can be when poorly designed).