-
Notifications
You must be signed in to change notification settings - Fork 59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Issue installing pydoop in Amazon emr-4.7.0 #214
Comments
Hi. It's not finding the JDK. Make sure it's installed. If it is, make sure Pydoop can find it by setting the JAVA_HOME environment variable. HTH Luca |
The JAVA_HOME is correctly set. I tried as a solution to install jdk 8.0 with: ` sudo yum -y install jdk-8-linux-x64.rpm NR_OF_OPTIONS=$(echo 0 | alternatives --config java 2>/dev/null | grep 'There ' | awk '{print $3}' | tail -1) sudo alternatives --install /usr/bin/java java /usr/java/default/bin/java 1 echo $(($NR_OF_OPTIONS + 1)) | sudo alternatives --config java export JAVA_HOME=/usr/java/default/bin/java export JRE_HOME=/usr/java/default/jre export PATH=$PATH:/usr/java/default/bin and after that I got: `
` |
so no one is able to solve this hadoop classpath issue? |
Hi there. Unfortunately we're out of AWS credits. Why don't you try your solution and submit a PR if it works? We'd be happy to receive your contribution. Luca |
@ilveroluca The reason I'm trying to install pydoop on EMR is because of TensorflowOnSpark. Fortunately, it is no longer a requirement. So I don't need it anymore. :) |
Checking now with emr-5.12.0. Pydoop needs to compile native code, so it needs the JDK home, i.e.: This is not the end of the story, though, since the logic that detects Java dependencies is not EMR-aware yet, so the Java code won't compile. I will update this ticket after fixing Java compilation. |
I have tried several approaches but it is not possible to install pydoop in Amazon emr-4.7.0. This is the command output:
pip install pydoop
You are using pip version 6.1.1, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pydoop
Downloading pydoop-1.2.0.tar.gz (956kB)
100% |████████████████████████████████| 958kB 442kB/s
Requirement already satisfied (use --upgrade to upgrade): setuptools>=3.3 in /usr/lib/python2.7/dist-packages (from pydoop)
Installing collected packages: pydoop
Running setup.py install for pydoop
Complete output from command /usr/bin/python2.7 -c "import setuptools, tokenize;file='/mnt/tmp/pip-build-X4tBci/pydoop/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-DwjTNG-record/install-record.txt --single-version-externally-managed --compile:
using setuptools version 12.2
running install
running build
hdfs core implementation: native
running build_py
creating build
creating build/lib
creating build/lib/pydoop
copying pydoop/test_support.py -> build/lib/pydoop
copying pydoop/init.py -> build/lib/pydoop
copying pydoop/pipes.py -> build/lib/pydoop
copying pydoop/hadoop_utils.py -> build/lib/pydoop
copying pydoop/jc.py -> build/lib/pydoop
copying pydoop/avrolib.py -> build/lib/pydoop
copying pydoop/test_utils.py -> build/lib/pydoop
copying pydoop/hadut.py -> build/lib/pydoop
copying pydoop/config.py -> build/lib/pydoop
copying pydoop/version.py -> build/lib/pydoop
creating build/lib/pydoop/mapreduce
copying pydoop/mapreduce/init.py -> build/lib/pydoop/mapreduce
copying pydoop/mapreduce/api.py -> build/lib/pydoop/mapreduce
copying pydoop/mapreduce/connections.py -> build/lib/pydoop/mapreduce
copying pydoop/mapreduce/pipes.py -> build/lib/pydoop/mapreduce
copying pydoop/mapreduce/streams.py -> build/lib/pydoop/mapreduce
copying pydoop/mapreduce/simulator.py -> build/lib/pydoop/mapreduce
copying pydoop/mapreduce/string_utils.py -> build/lib/pydoop/mapreduce
copying pydoop/mapreduce/binary_streams.py -> build/lib/pydoop/mapreduce
copying pydoop/mapreduce/jwritable_utils.py -> build/lib/pydoop/mapreduce
copying pydoop/mapreduce/text_streams.py -> build/lib/pydoop/mapreduce
creating build/lib/pydoop/utils
copying pydoop/utils/init.py -> build/lib/pydoop/utils
copying pydoop/utils/jvm.py -> build/lib/pydoop/utils
copying pydoop/utils/serialize.py -> build/lib/pydoop/utils
copying pydoop/utils/conversion_tables.py -> build/lib/pydoop/utils
copying pydoop/utils/misc.py -> build/lib/pydoop/utils
creating build/lib/pydoop/hdfs
copying pydoop/hdfs/path.py -> build/lib/pydoop/hdfs
copying pydoop/hdfs/init.py -> build/lib/pydoop/hdfs
copying pydoop/hdfs/fs.py -> build/lib/pydoop/hdfs
copying pydoop/hdfs/common.py -> build/lib/pydoop/hdfs
copying pydoop/hdfs/file.py -> build/lib/pydoop/hdfs
creating build/lib/pydoop/app
copying pydoop/app/init.py -> build/lib/pydoop/app
copying pydoop/app/script_template.py -> build/lib/pydoop/app
copying pydoop/app/submit.py -> build/lib/pydoop/app
copying pydoop/app/script.py -> build/lib/pydoop/app
copying pydoop/app/main.py -> build/lib/pydoop/app
copying pydoop/app/argparse_types.py -> build/lib/pydoop/app
creating build/lib/pydoop/utils/bridge
copying pydoop/utils/bridge/pyjnius_loader.py -> build/lib/pydoop/utils/bridge
copying pydoop/utils/bridge/init.py -> build/lib/pydoop/utils/bridge
copying pydoop/utils/bridge/factory.py -> build/lib/pydoop/utils/bridge
copying pydoop/utils/bridge/jpype_loader.py -> build/lib/pydoop/utils/bridge
creating build/lib/pydoop/hdfs/core
copying pydoop/hdfs/core/init.py -> build/lib/pydoop/hdfs/core
copying pydoop/hdfs/core/api.py -> build/lib/pydoop/hdfs/core
copying pydoop/hdfs/core/impl.py -> build/lib/pydoop/hdfs/core
creating build/lib/pydoop/hdfs/core/bridged
copying pydoop/hdfs/core/bridged/hadoop.py -> build/lib/pydoop/hdfs/core/bridged
copying pydoop/hdfs/core/bridged/init.py -> build/lib/pydoop/hdfs/core/bridged
copying pydoop/hdfs/core/bridged/common.py -> build/lib/pydoop/hdfs/core/bridged
copying pydoop/pydoop.properties -> build/lib/pydoop
running build_ext
building 'pydoop.sercore' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/src
creating build/temp.linux-x86_64-2.7/src/serialize
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c src/serialize/protocol_codec.cc -o build/temp.linux-x86_64-2.7/src/serialize/protocol_codec.o -Wno-write-strings -O3
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c src/serialize/SerialUtils.cc -o build/temp.linux-x86_64-2.7/src/serialize/SerialUtils.o -Wno-write-strings -O3
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c src/serialize/StringUtils.cc -o build/temp.linux-x86_64-2.7/src/serialize/StringUtils.o -Wno-write-strings -O3
g++ -pthread -shared build/temp.linux-x86_64-2.7/src/serialize/protocol_codec.o build/temp.linux-x86_64-2.7/src/serialize/SerialUtils.o build/temp.linux-x86_64-2.7/src/serialize/StringUtils.o -L/usr/lib64 -lpython2.7 -o build/lib/pydoop/sercore.so
building 'pydoop.native_core_hdfs' extension
creating build/temp.linux-x86_64-2.7/src/libhdfsV2
creating build/temp.linux-x86_64-2.7/src/libhdfsV2/common
creating build/temp.linux-x86_64-2.7/src/libhdfsV2/os
creating build/temp.linux-x86_64-2.7/src/libhdfsV2/os/posix
creating build/temp.linux-x86_64-2.7/src/native_core_hdfs
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DHADOOP_LIBHDFS_V2=1 -I/etc/alternatives/jre/include -Inative/jni_include -I/etc/alternatives/jre/lib -I/etc/alternatives/jre/include/linux -Isrc/libhdfsV2 -Isrc/libhdfsV2/os/posix -I/usr/include/python2.7 -c src/libhdfsV2/exception.c -o build/temp.linux-x86_64-2.7/src/libhdfsV2/exception.o -Wno-write-strings
In file included from src/libhdfsV2/exception.c:19:0:
src/libhdfsV2/exception.h:39:17: fatal error: jni.h: No existe el fichero o el directorio
#include <jni.h>
^
compilation terminated.
error: command 'gcc' failed with exit status 1
The text was updated successfully, but these errors were encountered: