Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Update JobTest to call transformForRead on pipes. #297

Closed
wants to merge 1 commit into from

3 participants

@arkajit
Collaborator

I want JobTest to actually apply a transformForRead on it's sources. It doesn't seem to do this now in the CascadingLocal Test mode and that's foiling one of my job tests.

I haven't figured it out yet, but wanted to start a discussion on how to do this. My first attempt doesn't quite work, but it's a rough idea of what I'd like to be able to do. Thoughts?

@azymnis azymnis was assigned
@johnynek johnynek was assigned
@azymnis
Collaborator

See #298

Looks to me like JobTest is already calling transformForRead/transformForWrite since:
1. This test is passing
2. The code for this is in Source and not in JobTest
3. I verified it with some printf debugging

So the bug must be elsewhere. Let's try and hunt this down tomorrow in person.

@johnynek
Collaborator

Yeah, I tried to find how to get it called, but I couldn't convince myself it wasn't actually called. I think this may be a folk theorem, or a special case bug, not a general situation.

@johnynek johnynek closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Feb 2, 2013
  1. @arkajit
This page is out of date. Refresh to see the latest.
Showing with 2 additions and 0 deletions.
  1. +2 −0  src/main/scala/com/twitter/scalding/Mode.scala
View
2  src/main/scala/com/twitter/scalding/Mode.scala
@@ -134,6 +134,8 @@ trait CascadingLocal extends Mode {
val fp = new LocalFlowProcess
ltap.openForRead(fp)
}
+ // This is roughly what I want, but it doesn't work since transformForRead is protected.
+ // override def getReadPipe(s : Source, p: => Pipe) : Pipe = s.transformForRead(super.getReadPipe(s, p))
}
// Mix-in trait for test modes; overrides fileExists to allow the registration
Something went wrong with that request. Please try again.