[HUDI-4985] Update ARM64 Docker demo images to use latest Hudi version by deepakpanda93 · Pull Request #13902 · apache/hudi

deepakpanda93 · 2025-09-16T13:54:22Z

Change Logs

Update ARM64 Docker demo images to use latest Hudi version

Impact

None

Risk level (write none, low medium or high below)

none

Documentation Update

We will raise another PR to do the necessary document changes for docker demo.

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

rangareddy · 2025-09-16T15:54:21Z

docker/hoodie/hadoop/historyserver/Dockerfile

Due to https://issues.apache.org/jira/browse/HADOOP-16614 you have added libleveldbjni.so?

@deepakpanda93 what about the other PR?
#13829 (comment)
i wanted to continue discussing the issue with using official hadoop image instead of maintaining a custom one.

if you're moving to another PR, please make sure they are linked. the previous review would be lost because of raising this new PR.

Hello @xushiyan

For now, Hadoop 3.3.4 docker image is not available. For the Spark version we are working, the supported version is hadoop 3.3.4. This is the reason I am still on the base image approach.

The other PR was for Hudi 1.0.2 and I think according to Apache guidelines, it may not allow to push to 1.0.2 branch. So created this PR for master branch.

Hello @rangareddy

The Docker image runs on an AARCH64 platform, and the absence of a compatible libleveldbjni library was preventing the HistoryServer from starting. To resolve this, I added the appropriate libleveldbjni version that supports AARCH64.

Error:

ERROR applicationhistoryservice.ApplicationHistoryServer: Error starting ApplicationHistoryServer java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni64-1.8 in java.library.path, no leveldbjni-1.8 in java.library.path, no leveldbjni in java.library.path, /tmp/libleveldbjni-64-1-8976797200427282087.8: /tmp/libleveldbjni-64-1-8976797200427282087.8: cannot open shared object file: No such file or directory (Possible cause: can't load AMD 64-bit .so on a AARCH64-bit platform)]

xushiyan

Please confirm the plan as commented, and fix the hive sync script issue. then we're good to go

xushiyan · 2025-09-17T17:33:03Z

hudi-sync/hudi-hive-sync/run_sync_tool.sh

  HIVE_JDBC=`ls ${HIVE_HOME}/lib/hive-jdbc-*.jar | grep -v handler | tr '\n' ':'`
 fi
-HIVE_JARS=$HIVE_METASTORE:$HIVE_SERVICE:$HIVE_EXEC:$HIVE_JDBC
+HIVE_JARS=$HIVE_METASTORE:$HIVE_SERVICE:$HIVE_EXEC:$HIVE_JDBC:${HIVE_HOME}/lib/calcite-core-1.16.0.jar:${HIVE_HOME}/lib/libfb303-0.9.3.jar


this could break if jars not available? how to ensure when user runs this script, jars are there? you're fix this because of the docker setup change right, but this script is not only intended for docker demo. you'll need to figure some way to decouple this

@deepakpanda93 this is the only blocking issue for this pr

@xushiyan The jars like calcite-core-1.16.0.jar and libfb303-0.9.3.jar is already available in the docker from which we added in the HiveSyncTool Classpath to make it work without dependency errors.
No manual effort is needed to fetch the required jars, they are auto-resolved by the build system.

i meant this script is not only used within docker demo. users can run it as standalone job for hive sync. Outside of docker demo, how can these 2 jars be always available? please have Aditya review PRs and give your guidance when in doubt.

xushiyan · 2025-09-17T17:39:56Z

docker/hoodie/hadoop/spark_base/Dockerfile

-ARG HADOOP_VERSION=2.8.4
-ARG HIVE_VERSION=2.3.3
+ARG HADOOP_VERSION=3.3.4
+ARG HIVE_VERSION=3.1.3


so to confirm, this pr is the first step which makes these changes:

make the same docker stack as the current docker demo, make sure it works for both amd64 and arm64

hadoop upgrade to 3.3.4, hive upgrade to 3.1.3

users still need to follow the same step to run the demo apps, like first build bundle jars, then specify this yml for the docker compose to run

The next step is add notebook container into the stack to allow users to use UI to work with the stack.

Then we need to avoid building jars and simplify the hadoop and other setup to keep the demo lightweight. @deepakpanda93 @rangareddy please confirm this is the plan to go with.

Yes

We are planning a few enhancements for the upcoming releases:

Update the Hudi Docker demo steps on the official Hudi website with clearer, more user-friendly instructions.

Add Jupyter notebooks to the Docker setup for easier experimentation and tutorials.

Test compatibility on both Docker amd64 and arm64 architectures to ensure broader platform support.

Adopt Apache’s official Docker images instead of maintaining custom-built ones.

Use Hudi jars from Maven Central rather than relying on locally built artifacts, ensuring consistency and reproducibility.

hudi-bot · 2025-09-18T18:12:39Z

CI report:

29e8bc1 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

xushiyan · 2025-09-19T01:29:21Z

docker/setup_demo.sh

 HUDI_DEMO_ENV=$1
 WS_ROOT=`dirname $SCRIPT_PATH`
 COMPOSE_FILE_NAME="docker-compose_hadoop284_hive233_spark353_amd64.yml"
-if [ "$HUDI_DEMO_ENV" = "--mac-aarch64" ]; then
-  COMPOSE_FILE_NAME="docker-compose_hadoop284_hive233_spark353_arm64.yml"
+if [ "$(uname -m)" = "arm64" ]; then


@deepakpanda93 in follow up pr, clean up this HUDI_DEMO_ENV as you made it unused. Also update any docs (search in branch asf-site) that shows how to use this script accordingly.

xushiyan · 2025-09-19T01:31:09Z

docker/stop_demo.sh

 # set up root directory
 WS_ROOT=`dirname $SCRIPT_PATH`
 COMPOSE_FILE_NAME="docker-compose_hadoop284_hive233_spark353_amd64.yml"
-if [ "$HUDI_DEMO_ENV" = "--mac-aarch64" ]; then


same as the setup demo script

[HUDI-4985] Update ARM64 Docker demo images to use latest Hudi version

5bcffa1

github-actions bot added the size:L PR with lines of changes in (300, 1000] label Sep 16, 2025

rangareddy reviewed Sep 16, 2025

View reviewed changes

deepakpanda93 mentioned this pull request Sep 16, 2025

[HUDI-4985] Update ARM64 Docker demo images to use latest Hudi version #13829

Closed

4 tasks

xushiyan reviewed Sep 17, 2025

View reviewed changes

Update run_sync_tool.sh

29e8bc1

xushiyan approved these changes Sep 19, 2025

View reviewed changes

xushiyan merged commit e3e8b73 into apache:master Sep 19, 2025
61 of 62 checks passed

This was referenced Sep 19, 2025

fix: update docker compose file name for mac arm in the stop script #13928

Closed

stop docker demo mac has wrong compose file name #13929

Closed

yihua mentioned this pull request Dec 8, 2025

Update the arm64 images in docker demo to the latest hudi version #15482

Closed

Conversation

deepakpanda93 commented Sep 16, 2025

Change Logs

Impact

Risk level (write none, low medium or high below)

Documentation Update

Contributor's checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xushiyan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rangareddy Sep 17, 2025 • edited by xushiyan Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hudi-bot commented Sep 18, 2025

CI report:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rangareddy Sep 17, 2025 •

edited by xushiyan

Loading