Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to Debian 10, Java 11, Hadoop 3.3.1, build/run on ARM #108

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

sebastian-nagel
Copy link

$> hadoop checknative
2021-07-02 15:22:55,211 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
2021-07-02 15:22:55,214 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2021-07-02 15:22:55,218 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
2021-07-02 15:22:55,253 INFO nativeio.NativeIO: The native code was built without PMDK support.
Native library checking:
hadoop:  true /opt/hadoop-3.3.1/lib/native/libhadoop.so.1.0.0
zlib:    true /lib/aarch64-linux-gnu/libz.so.1
zstd  :  true /usr/lib/aarch64-linux-gnu/libzstd.so.1
bzip2:   true /lib/aarch64-linux-gnu/libbz2.so.1
openssl: true /usr/lib/aarch64-linux-gnu/libcrypto.so
ISA-L:   false libhadoop was built without ISA-L support
PMDK:    false The native code was built without PMDK support.

$> hadoop version
Hadoop 3.3.1
Source code repository https://github.com/apache/hadoop.git -r a3b9c37a397ad4188041dd80621bdeefc46885f2
Compiled by ubuntu on 2021-06-15T10:51Z
Compiled with protoc 3.7.1
From source with checksum 88a4ddb2299aca054416d6b7f81ca55
This command was run using /opt/hadoop-3.3.1/share/hadoop/common/hadoop-common-3.3.1.jar

- pass tag/version of hadoop-base image forward to Docker
  builds of components (namenode, etc.) so that all newly built
  component images are based on the same hadoop-base image
by setting a softlink /opt/hadoop/ pointing to the versioned
Hadoop installation directory
manager and history server). Internal Hadoop communication ports
are reachable only from inside the cluster.

Fix Docker network name in README.md - should be the same as defined
by DOCKER_NETWORK in the Makefile.
- allow to build/run on ARM architecture (aarch64 / arm64)
- include libraries required by Hadoop native libs
@jacobmstein
Copy link

jacobmstein commented May 11, 2022

I spent hours going through and fixing all these issues to try and run this on my M1 Mac. Finally found your PR, it fixes everything I missed. Thank you!

edit: Please note, this does not work on Raspberry Pi 4 with Ubuntu Server. I will get more details shortly, however it selects a non-existent build even though it's ARMv8. I had to modify the build selection logic.

@OneCricketeer
Copy link

Hadoop 3.3.3 officially supports Java 11 - https://hadoop.apache.org/docs/current/

@lemonGGGit
Copy link

I spent hours going through and fixing all these issues to try and run this on my M1 Mac. Finally found your PR, it fixes everything I missed. Thank you!

edit: Please note, this does not work on Raspberry Pi 4 with Ubuntu Server. I will get more details shortly, however it selects a non-existent build even though it's ARMv8. I had to modify the build selection logic.

@jacobmstein, Hello, Can you work it on 「Raspberry Pi 4 with Ubuntu Server」 now?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants