Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Dumbo doesn't work with CDH4b2 nightlies #53

Closed
aripollak opened this Issue Apr 19, 2012 · 5 comments

Comments

Projects
None yet
3 participants

Starting with CDH4b2 (nightlies at http://nightly.cloudera.com/cdh4/), /usr/lib/hadoop has been split out into a bunch of different directories, like hadoop-hdfs and hadoop-mapreduce, so a lot of the assumptions made in dumbo no longer work.
A few examples:

  • If I don't set JAVA_HOME=/usr/lib/jvm/default-java and HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec, I can't even run dumbo ls / -hadoop /usr/lib/hadoop since it will complain about JAVA_HOME not being set or not being able to find libexec.
  • Trying to run dumbo start wordcount.py -input foo -output bar -hadoop /usr/lib/hadoop results in "ERROR: Streaming jar not found" since the streaming jar is now under /usr/lib/hadoop-mapreduce.

I might be able to fix this if I find some free time, but it might require some structural changes.

Owner

klbostee commented Apr 21, 2012

Right, sounds annoying.

Could we fix this by extending the -hadoop option to take a list of (comma separated) directories maybe? You can define aliases in /etc/dumbo.conf, so we could fairly easily avoid having to give it that list all the time...

That could be an option, at least for the immediate problem of looking for JARs in the right place. Or it could try automatically looking in <hadoop_dir>-mapreduce instead of just hadoop_dir?

Owner

klbostee commented Apr 21, 2012

Guess that could work too yeah. It might even be worth implementing both maybe, so that people with other distributions have enough flexibility to get things to work as well...

Is there a workaround available for this ? I have a fresh installation of latest Dumbo and Hadoop Cloudera 2.0.0 CDH4 and am still seeing this error. Also, tried linking all jars in the various folder into one folder and use that, no luck.

Any hints would be very helpful

Owner

klbostee commented Aug 21, 2012

See comments on the pull request...

@klbostee klbostee closed this Aug 21, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment