-
Notifications
You must be signed in to change notification settings - Fork 705
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature/bijected source sink #886
Conversation
|
||
object BijectedSourceSink { | ||
type SourceSink[T] = TypedSource[T] with TypedSink[T] | ||
def apply[T, U](parent: SourceSink[T], transformer: Bijection[T, U]) = new BijectedSourceSink(parent, transformer) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should transformer
here be implicit?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thats a good question, no particular reason why not... will update
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated as an implicit, and updated the tests with the cleaner workflow that results
def apply[T, U](parent: SourceSink[T])(implicit transformer: Bijection[T, U]) = new BijectedSourceSink(parent)(transformer) | ||
} | ||
|
||
class BijectedSourceSink[T, U](parent: BijectedSourceSink.SourceSink[T])(implicit transformer: Bijection[T, U]) extends TypedSource[U] with TypedSink[U] { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
extends BijectedSourceSink.SourceSink[U]
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
had tried that, it didn't like extending off the class defined type for some reason
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks awesome. I wonder if implicit transformer: Bijection[T, U])
would have serialization issues or should be lazy?
On Thu, May 29, 2014 at 1:09 PM, ianoc notifications@github.com wrote:
In
scalding-core/src/main/scala/com/twitter/scalding/typed/BijectedSourceSink.scala:+*/
+package com.twitter.scalding.typed
+
+import cascading.flow.FlowDef
+import cascading.pipe.Pipe
+
+import com.twitter.bijection.Bijection
+import com.twitter.scalding._
+
+
+object BijectedSourceSink {
- type SourceSink[T] = TypedSource[T] with TypedSink[T]
- def apply[T, U](parent: SourceSink[T])(implicit transformer: Bijection[T, U]) = new BijectedSourceSink(parent)(transformer)
+}
+class BijectedSourceSink[T, U](parent: BijectedSourceSink.SourceSink[T])(implicit transformer: Bijection[T, U]) extends TypedSource[U] with TypedSink[U] {
had tried that, it didn't like extending off the class defined type for
some reason—
Reply to this email directly or view it on GitHub
https://github.com/twitter/scalding/pull/886/files#r13195197.
tight! |
Add a bijected source sink which allows composing a bijection with a typedsink
|
||
object BijectedSourceSink { | ||
type SourceSink[T] = TypedSource[T] with TypedSink[T] | ||
def apply[T, U](parent: SourceSink[T])(implicit transformer: Bijection[T, U]) = new BijectedSourceSink(parent)(transformer) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you should take ImplicitBijection
here, not Bijection. This is a trick to deal with the fact that Bijection[A, B]
and Bijection[B, A]
are equivalent, but confuse implicit resolution.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
type on a public method.
Use case is that you have a Source on disk that you want to treat as a different type and have a bijection between those types. Allows reading/writing through the bijection