-
Notifications
You must be signed in to change notification settings - Fork 339
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added BatchSourceContext.setInput(InputFormatProvider) #4512
Conversation
chtyim
commented
Nov 9, 2015
- Remove usage of BatchContext.getHadoopJob
- Also move MapReduce classes under "mapreduce" package
0b7277c
to
88f7f4c
Compare
88f7f4c
to
2467d13
Compare
config.put(entry.getKey(), entry.getValue()); | ||
} | ||
} catch (IOException e) { | ||
// Shouldn't happen |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't happen because tpfsAvroConfig.schema
has been validated before in configurePipeline
method?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The IOException is from the Job.getInstance()
call.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh ok. 👍
One question. Rest LGTM 👍 |
Configuration hConf = job.getConfiguration(); | ||
hConf.clear(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why is this needed? So the hConf
passed to SourceInputFormatProvider
on line 101 does not have anything other than DBConfiguration
, ETLDBInputFormat
properties?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
otherwise it'll have everything from the core-default.xml
, which are unnecessary.
9c2efc8
to
6a70f8d
Compare
Build passed: http://builds.cask.co/browse/CDAP-DUT3117-3 |
- Remove usage of BatchContext.getHadoopJob - Also move MapReduce classes under "mapreduce" package
6a70f8d
to
d6f3a56
Compare
Squashed commits. Merging. |
…usage Added BatchSourceContext.setInput(InputFormatProvider)