Feature/bijected source sink #886
Conversation
|
|
||
| object BijectedSourceSink { | ||
| type SourceSink[T] = TypedSource[T] with TypedSink[T] | ||
| def apply[T, U](parent: SourceSink[T], transformer: Bijection[T, U]) = new BijectedSourceSink(parent, transformer) |
There was a problem hiding this comment.
Should transformer here be implicit?
There was a problem hiding this comment.
Thats a good question, no particular reason why not... will update
There was a problem hiding this comment.
Updated as an implicit, and updated the tests with the cleaner workflow that results
| def apply[T, U](parent: SourceSink[T])(implicit transformer: Bijection[T, U]) = new BijectedSourceSink(parent)(transformer) | ||
| } | ||
|
|
||
| class BijectedSourceSink[T, U](parent: BijectedSourceSink.SourceSink[T])(implicit transformer: Bijection[T, U]) extends TypedSource[U] with TypedSink[U] { |
There was a problem hiding this comment.
extends BijectedSourceSink.SourceSink[U]?
There was a problem hiding this comment.
had tried that, it didn't like extending off the class defined type for some reason
There was a problem hiding this comment.
This looks awesome. I wonder if implicit transformer: Bijection[T, U])
would have serialization issues or should be lazy?
On Thu, May 29, 2014 at 1:09 PM, ianoc notifications@github.com wrote:
In
scalding-core/src/main/scala/com/twitter/scalding/typed/BijectedSourceSink.scala:+*/
+package com.twitter.scalding.typed
+
+import cascading.flow.FlowDef
+import cascading.pipe.Pipe
+
+import com.twitter.bijection.Bijection
+import com.twitter.scalding._
+
+
+object BijectedSourceSink {
- type SourceSink[T] = TypedSource[T] with TypedSink[T]
- def apply[T, U](parent: SourceSink[T])(implicit transformer: Bijection[T, U]) = new BijectedSourceSink(parent)(transformer)
+}
+class BijectedSourceSink[T, U](parent: BijectedSourceSink.SourceSink[T])(implicit transformer: Bijection[T, U]) extends TypedSource[U] with TypedSink[U] {
had tried that, it didn't like extending off the class defined type for
some reason—
Reply to this email directly or view it on GitHub
https://github.com/twitter/scalding/pull/886/files#r13195197.
|
tight! |
Add a bijected source sink which allows composing a bijection with a typedsink
|
|
||
| object BijectedSourceSink { | ||
| type SourceSink[T] = TypedSource[T] with TypedSink[T] | ||
| def apply[T, U](parent: SourceSink[T])(implicit transformer: Bijection[T, U]) = new BijectedSourceSink(parent)(transformer) |
There was a problem hiding this comment.
you should take ImplicitBijection here, not Bijection. This is a trick to deal with the fact that Bijection[A, B] and Bijection[B, A] are equivalent, but confuse implicit resolution.
There was a problem hiding this comment.
type on a public method.
Use case is that you have a Source on disk that you want to treat as a different type and have a bijection between those types. Allows reading/writing through the bijection