Skip to content

Change image hudi-hadoop_2.8.4-history version to linux-arm64-0.10.1 adapt to MacOS M1#8999

Closed
xiedeyantu wants to merge 1 commit intoapache:masterfrom
xiedeyantu:patch-1
Closed

Change image hudi-hadoop_2.8.4-history version to linux-arm64-0.10.1 adapt to MacOS M1#8999
xiedeyantu wants to merge 1 commit intoapache:masterfrom
xiedeyantu:patch-1

Conversation

@xiedeyantu
Copy link
Member

@xiedeyantu xiedeyantu commented Jun 16, 2023

Change Logs

Change image hudi-hadoop_2.8.4-history version to linux-arm64-0.10.1 adapt to MacOS M1

Impact

Describe any public API or user-facing feature change or any performance impact.

Risk level (write none, low medium or high below)

If medium or high, explain what verification was done to mitigate the risks.

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change

  • The config description must be updated if new configs are added or the default value of the configs are changed
  • Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
    ticket number here and follow the instruction to make
    changes to the website.

Contributor's checklist

  • Read through contributor's guide
  • Change Logs and Impact were stated clearly
  • Adequate tests were added if applicable
  • CI passed

@hudi-bot
Copy link
Collaborator

hudi-bot commented Jun 16, 2023

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@yihua
Copy link
Contributor

yihua commented Jun 16, 2023

@jonvex could you review this?

@yihua yihua added the component:docker Docker-related label Jun 16, 2023
@jonvex
Copy link
Contributor

jonvex commented Jun 16, 2023

When I tested this, The history server fails with the exception:

2023-06-16 15:14:34 ************************************************************/
2023-06-16 15:14:34 23/06/16 19:14:34 INFO applicationhistoryservice.ApplicationHistoryServer: registered UNIX signal handlers for [TERM, HUP, INT]
2023-06-16 15:14:35 23/06/16 19:14:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2023-06-16 15:14:35 23/06/16 19:14:35 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2023-06-16 15:14:35 23/06/16 19:14:35 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2023-06-16 15:14:35 23/06/16 19:14:35 INFO impl.MetricsSystemImpl: ApplicationHistoryServer metrics system started
2023-06-16 15:14:35 23/06/16 19:14:35 FATAL applicationhistoryservice.ApplicationHistoryServer: Error starting ApplicationHistoryServer
2023-06-16 15:14:35 java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni64-1.8 in java.library.path, no leveldbjni-1.8 in java.library.path, no leveldbjni in java.library.path, /tmp/libleveldbjni-64-1-2504575729329802225.8: /tmp/libleveldbjni-64-1-2504575729329802225.8: cannot open shared object file: No such file or directory (Possible cause: can't load AMD 64-bit .so on a AARCH64-bit platform)]
2023-06-16 15:14:35     at org.fusesource.hawtjni.runtime.Library.doLoad(Library.java:182)
2023-06-16 15:14:35     at org.fusesource.hawtjni.runtime.Library.load(Library.java:140)
2023-06-16 15:14:35     at org.fusesource.leveldbjni.JniDBFactory.<clinit>(JniDBFactory.java:48)
2023-06-16 15:14:35     at org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore.serviceInit(LeveldbTimelineStore.java:227)
2023-06-16 15:14:35     at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
2023-06-16 15:14:35     at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
2023-06-16 15:14:35     at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:115)
2023-06-16 15:14:35     at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
2023-06-16 15:14:35     at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:180)
2023-06-16 15:14:35     at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:190)
2023-06-16 15:14:35 23/06/16 19:14:35 INFO util.ExitUtil: Exiting with status -1

@xiedeyantu
Copy link
Member Author

xiedeyantu commented Jun 17, 2023

When I tested this, The history server fails with the exception:

2023-06-16 15:14:34 ************************************************************/
2023-06-16 15:14:34 23/06/16 19:14:34 INFO applicationhistoryservice.ApplicationHistoryServer: registered UNIX signal handlers for [TERM, HUP, INT]
2023-06-16 15:14:35 23/06/16 19:14:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2023-06-16 15:14:35 23/06/16 19:14:35 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2023-06-16 15:14:35 23/06/16 19:14:35 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2023-06-16 15:14:35 23/06/16 19:14:35 INFO impl.MetricsSystemImpl: ApplicationHistoryServer metrics system started
2023-06-16 15:14:35 23/06/16 19:14:35 FATAL applicationhistoryservice.ApplicationHistoryServer: Error starting ApplicationHistoryServer
2023-06-16 15:14:35 java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni64-1.8 in java.library.path, no leveldbjni-1.8 in java.library.path, no leveldbjni in java.library.path, /tmp/libleveldbjni-64-1-2504575729329802225.8: /tmp/libleveldbjni-64-1-2504575729329802225.8: cannot open shared object file: No such file or directory (Possible cause: can't load AMD 64-bit .so on a AARCH64-bit platform)]
2023-06-16 15:14:35     at org.fusesource.hawtjni.runtime.Library.doLoad(Library.java:182)
2023-06-16 15:14:35     at org.fusesource.hawtjni.runtime.Library.load(Library.java:140)
2023-06-16 15:14:35     at org.fusesource.leveldbjni.JniDBFactory.<clinit>(JniDBFactory.java:48)
2023-06-16 15:14:35     at org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore.serviceInit(LeveldbTimelineStore.java:227)
2023-06-16 15:14:35     at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
2023-06-16 15:14:35     at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
2023-06-16 15:14:35     at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:115)
2023-06-16 15:14:35     at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
2023-06-16 15:14:35     at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:180)
2023-06-16 15:14:35     at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:190)
2023-06-16 15:14:35 23/06/16 19:14:35 INFO util.ExitUtil: Exiting with status -1

The problem with the image itself requires me to recreate the image through Mac m1, but I do not have the permission to create an image and push to apachehudi registry. I can change all the images to my own registry. which is a batter way?

@jonvex
Copy link
Contributor

jonvex commented Jun 18, 2023

If I recall, when we created the images the historyserver wasn't working with linux-arm64-0.10.1. But you tested in your own registry and it now works? I'll try to update it in the next couple of days

@xiedeyantu
Copy link
Member Author

xiedeyantu commented Jun 18, 2023

Yes, I rebuild historyserver image, it works. If you can update it, just close my pr please.

@xiedeyantu xiedeyantu closed this Aug 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@jonvex jonvex

Labels

component:docker Docker-related

Projects

Hudi PR Support
Status: ✅ Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@xiedeyantu @hudi-bot @yihua @jonvex