No support for ARM based architectures (Missing hadoop native lib)

[PostProd]

Currently, there prestodb doesn't support ARM architectures for two reasons. The first is the PrestoSystemRequirements class which doesn't include "aarch64" in the allowed os.arch values. The second is that the underlying dependency for our hadoop jars is from 'com.facebook.presto.hadoop' which doesn't currently include nativelibs/ for the aarch64 ARM types. This leaves two options:

Option 1: Have Facebook update their hadoop libraries to also build nativelibs/ for "aarch64"
Option 2: Change the dependency on hadoop packaging to be some other package that does bundle nativelibs/ for "aarch64"

[rohanpednekar]

Hi @v-jizhang here is another user request on the same topic we discussed last week. Do you think you can take it as a priority? Thanks!

[rohanpednekar]

Discussed this with @imjalpreet,

the Hadoop dependency being used in Presto is a custom shaded version of Apache Hadoop. It is also open source and can be worked on updating to Hadoop 3 as well as adding the required Hadoop native libraries for arm. Custom shaded version is done to include only the required dependencies so that the size is not too big. So, Option 2 is not possible. The only way is to modify the current dependency to make it compatible for ARM.

[v-jizhang]

• Update to Haddop3
  We already have a pull request PR53

• Add support for Linux aarch64
  Cherry-pick of Add support for Linux aarch64.

• Add support for macOS aarch64
  Cherry-pick of Add support for macOS aarch64

[PostProd]

Why is Hadoop 3 a requirement for native ARM libraries? Is it not feasible to recompile the native libs on Hadoop 2.x? Additionally, if we were to move to Hadoop 3, why not consider Hadoop 3.3.x since it upgrades the protobuf dependency from 2.5.0 to 3.7.1? This adds significant value since protobuf 2.5.0 is unsupported/deprecated and doesn't even build Hadoop libs out of box (requires headers for atomics).

Hadoop 3.3.0 Release Notes: https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-common/release/3.3.0/RELEASENOTES.3.3.0.html

[vbvasa]

@v-jizhang - It would be great if we can take this on priority. We would like to move our Presto On Spark clusters to AWS Graviton and are waiting for presto to support ARM64.
Thanks in advance!

[Akanksha-kedia]

i m still facing issue chip : Apple M1 Pro but issue :
java.lang.ExceptionInInitializerError
at io.prestosql.plugin.hive.HdfsEnvironment$$FastClassByGuice$$2c8553d4.newInstance()
at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
at com.google.inject.internal.ConstructorInjector.access$000(ConstructorInjector.java:32)
at com.google.inject.internal.ConstructorInjector$1.call(ConstructorInjector.java:98)
at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:112)
at io.airlift.bootstrap.LifeCycleModule.provision(LifeCycleModule.java:54)
at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:120)
at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:93)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
at com.google.inject.internal.ConstructorInjector.access$000(ConstructorInjector.java:32)
at com.google.inject.internal.ConstructorInjector$1.call(ConstructorInjector.java:98)
at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:112)
at io.airlift.bootstrap.LifeCycleModule.provision(LifeCycleModule.java:54)
at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:120)
at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:93)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:213)
at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:184)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:111)
at com.google.inject.Guice.createInjector(Guice.java:87)
at io.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:276)
at io.prestosql.plugin.hive.InternalHiveConnectorFactory.createConnector(InternalHiveConnectorFactory.java:117)
at io.prestosql.plugin.hive.InternalHiveConnectorFactory.createConnector(InternalHiveConnectorFactory.java:77)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at io.prestosql.plugin.hive.HiveConnectorFactory.create(HiveConnectorFactory.java:63)
at io.prestosql.connector.ConnectorManager.createConnector(ConnectorManager.java:353)
at io.prestosql.connector.ConnectorManager.createCatalog(ConnectorManager.java:212)
at io.prestosql.connector.ConnectorManager.createCatalog(ConnectorManager.java:204)
at io.prestosql.connector.ConnectorManager.createCatalog(ConnectorManager.java:190)
at io.prestosql.metadata.StaticCatalogStore.loadCatalog(StaticCatalogStore.java:88)
at io.prestosql.metadata.StaticCatalogStore.loadCatalogs(StaticCatalogStore.java:68)
at io.prestosql.server.Server.doStart(Server.java:119)
at io.prestosql.server.Server.lambda$start$0(Server.java:73)
at io.prestosql.$gen.Presto_unknown____20240504_042839_1.run(Unknown Source)
at io.prestosql.server.Server.start(Server.java:73)
at io.prestosql.server.PrestoServer.main(PrestoServer.java:38)
Caused by: java.lang.RuntimeException: failed to load Hadoop native library
at io.prestosql.hadoop.HadoopNative.requireHadoopNative(HadoopNative.java:59)
at io.prestosql.plugin.hive.HdfsEnvironment.(HdfsEnvironment.java:39)
... 52 more
Caused by: java.lang.RuntimeException: library not found: /nativelib/Mac_OS_X-aarch64/libhadoop.dylib
at io.prestosql.hadoop.HadoopNative.loadLibrary(HadoopNative.java:89)
at io.prestosql.hadoop.HadoopNative.requireHadoopNative(HadoopNative.java:47)
... 53 more

Screenshot 2024-05-04 at 10 01 10 AM
also i tried exporting export DYLD_LIBRARY_PATH=/path/

Downloaded the library from the provided link and saved it to a suitable directory, say /Users/ak/lib.

Set the DYLD_LIBRARY_PATH environment variable to include the directory where saved the libhadoop.dylib file:

export DYLD_LIBRARY_PATH=/Users/ak/lib