Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Missing dependency or reference to hadoop library or binary (Windows) #20

Closed
eugenechung81 opened this issue May 11, 2015 · 3 comments
Closed

Missing dependency or reference to hadoop library or binary (Windows) #20

eugenechung81 opened this issue May 11, 2015 · 3 comments

Comments

@eugenechung81
Copy link

Running sbt on windows environment results in the following:

$ sbt app/run

[ERROR] [2015-05-11 14:25:17,117] [org.apache.hadoop.util.Shell]: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

Log output:

[INFO] [2015-05-11 14:25:14,524] [Remoting]: Starting remoting
[INFO] [2015-05-11 14:25:14,535] [Remoting]: Remoting started; listening on addr
esses :[akka.tcp://sparkDriver@172.16.3.2:55917]
[INFO] [2015-05-11 14:25:15,642] [com.datastax.killrweather.NodeGuardian]: Start
ing at akka.tcp://KillrWeather@127.0.0.1:2550
[INFO] [2015-05-11 14:25:15,653] [Cluster(akka://KillrWeather)]: Cluster Node [a
kka.tcp://KillrWeather@127.0.0.1:2550] - Node [akka.tcp://KillrWeather@127.0.0.1
:2550] is JOINING, roles []
[INFO] [2015-05-11 14:25:16,431] [Cluster(akka://KillrWeather)]: Cluster Node [a
kka.tcp://KillrWeather@127.0.0.1:2550] - Leader is moving node [akka.tcp://Killr
Weather@127.0.0.1:2550] to [Up]
[INFO] [2015-05-11 14:25:16,433] [com.datastax.killrweather.NodeGuardian]: Membe
r akka.tcp://KillrWeather@127.0.0.1:2550 joined cluster.
[INFO] [2015-05-11 14:25:16,498] [com.datastax.killrweather.NodeGuardian]: Node

is transitioning from 'uninitialized' to 'initialized'

Time: 1431368717000 ms

[ERROR] [2015-05-11 14:25:17,117] [org.apache.hadoop.util.Shell]: Failed to loca
te the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Ha
doop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:278) [had
oop-common-2.2.0.jar:na]
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:300) [hadoop-
common-2.2.0.jar:na]
at org.apache.hadoop.util.Shell.(Shell.java:293) [hadoop-common-
2.2.0.jar:na]
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSys
tem.java:639) [hadoop-common-2.2.0.jar:na]
at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.
java:468) [hadoop-common-2.2.0.jar:na]
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.jav
a:456) [hadoop-common-2.2.0.jar:na]
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.jav
a:424) [hadoop-common-2.2.0.jar:na]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:905) [hadoop-c
ommon-2.2.0.jar:na]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:886) [hadoop-c
ommon-2.2.0.jar:na]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:783) [hadoop-c
ommon-2.2.0.jar:na]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:772) [hadoop-c
ommon-2.2.0.jar:na]
at org.apache.spark.streaming.CheckpointWriter$CheckpointWriteHandler.ru
n(Checkpoint.scala:135) [spark-streaming_2.10-1.2.1.jar:1.2.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145) [na:1.7.0_51]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615) [na:1.7.0_51]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
error java.lang.NullPointerException
java.lang.NullPointerException
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1010)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:404)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:
589)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:678)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:661)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSys
tem.java:639)
at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.
java:468)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.jav
a:456)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.jav
a:424)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:905)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:886)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:783)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:772)
at org.apache.spark.streaming.CheckpointWriter$CheckpointWriteHandler.ru
n(Checkpoint.scala:135)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
at java.lang.Thread.run(Thread.java:744)
[trace] Stack trace suppressed: run last app/*:run for the full output.
error java.lang.NullPointerException
java.lang.NullPointerException
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1010)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:404)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:
589)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:678)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:661)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSys
tem.java:639)
at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.
java:468)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.jav
a:456)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.jav
a:424)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:905)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:886)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:783)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:772)
at org.apache.spark.streaming.CheckpointWriter$CheckpointWriteHandler.ru
n(Checkpoint.scala:135)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
at java.lang.Thread.run(Thread.java:744)

[trace] Stack trace suppressed: run last app/*:run for the full output.

Time: 1431368717500 ms
@DobrinGanev
Copy link

You may have to install hadoop first and then run the app.
http://stackoverflow.com/questions/19620642/failed-to-locate-the-winutils-binary-in-the-hadoop-binary-path
I have no problem running it on Windows 7-8.1. but I had hadoop already installed.
Actually I had all of them installed first. Spark, Cassandra, Scala, sbt.

@eugenechung81
Copy link
Author

Thanks, instead of building the binary from scratch for WindowOS, I just look snapshot of the common binaries here (https://github.com/srccodes/hadoop-common-2.2.0-bin/archive/master.zip) and it worked. I see the isd_weather_data tables being populated.

@heayin
Copy link

heayin commented May 28, 2015

hi eugenechung81:
how do you configure the hadoop-common binaries? I have the same problem. It is greatly appreciated for your help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@heayin @DobrinGanev @eugenechung81