Skip to content


Subversion checkout URL

You can clone with
Download ZIP


Update JobTest to call transformForRead on pipes. #297

wants to merge 1 commit into from

3 participants


I want JobTest to actually apply a transformForRead on it's sources. It doesn't seem to do this now in the CascadingLocal Test mode and that's foiling one of my job tests.

I haven't figured it out yet, but wanted to start a discussion on how to do this. My first attempt doesn't quite work, but it's a rough idea of what I'd like to be able to do. Thoughts?

@azymnis azymnis was assigned
@johnynek johnynek was assigned

See #298

Looks to me like JobTest is already calling transformForRead/transformForWrite since:
1. This test is passing
2. The code for this is in Source and not in JobTest
3. I verified it with some printf debugging

So the bug must be elsewhere. Let's try and hunt this down tomorrow in person.


Yeah, I tried to find how to get it called, but I couldn't convince myself it wasn't actually called. I think this may be a folk theorem, or a special case bug, not a general situation.

@johnynek johnynek closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Feb 2, 2013
  1. @arkajit
This page is out of date. Refresh to see the latest.
Showing with 2 additions and 0 deletions.
  1. +2 −0  src/main/scala/com/twitter/scalding/Mode.scala
2  src/main/scala/com/twitter/scalding/Mode.scala
@@ -134,6 +134,8 @@ trait CascadingLocal extends Mode {
val fp = new LocalFlowProcess
+ // This is roughly what I want, but it doesn't work since transformForRead is protected.
+ // override def getReadPipe(s : Source, p: => Pipe) : Pipe = s.transformForRead(super.getReadPipe(s, p))
// Mix-in trait for test modes; overrides fileExists to allow the registration
Something went wrong with that request. Please try again.