Bug fixes for Hive CLI/Beeline compatibility #1616
Conversation
|
Thanks this looks good.... Except for the ALL CAPS COMMIT MESSAGE WHICH WILL BE EXTREMELY DISTRACTING FOR PEOPLE LOOKING IN THE COMMIT LOG. Please amend you commit. After that this can be merged. :) |
when you use beeline instead of hive you can specify hive command in luigi config file butyour command may have multiple options. For this reason the load_hive_cmd() method should return an array and not a single string. When your task has parameters, [self.task_id] has parenthesis and this will cause shell.Popen issues (OsError). Fixed by wrapping [self.task_id] in quotes.
PeteW
changed the title
|
@Tarrasch THE COMMIT MESSAGE HAS BEEN AMENDED PER YOUR REQUEST. SORRY ABOUT THAT. I HAVE TROUBLE WITH TYPING LOUD SOMETIMES |
| @@ -297,7 +297,7 @@ def hiveconfs(self): | |||
| * hive.exec.reducers.max (reducers_max) | |||
| """ | |||
| jcs = {} | |||
| jcs['mapred.job.name'] = self.task_id | |||
| jcs['mapred.job.name'] = "'" + self.task_id + "'" | |||
There was a problem hiding this comment.
"'%s'" % self.task_id would have been marginally more pythonic, but not blocking
|
LOOKS GOOD TO ME |
|
Sorry to ask for another round-trip. But can you also remove the useless merge commit in this PR? It should only be 1 commit. |
|
Actually no need! I can now squash commits directly from the web browser! Finally!! 🎉 🎉 🎉 |
|
Cool! BTW I love your work. I also found more "issues" maybe bugs? maybe I am the On 4 April 2016 at 00:14, Arash Rouhani notifications@github.com wrote:
Peter Weissbrod |
|
Haha. :) Please send pull requests if you find any more bugs! That's the community spirit that keeps the code base up-to-date! |
when you use beeline instead of hive you can specify hive command in luigi config file butyour command may have multiple options. For this reason the load_hive_cmd() method should return an array and not a single string.
When your task has parameters, [self.task_id] has parenthesis and this will cause shell.Popen issues (OsError). Fixed by wrapping [self.task_id] in quotes.