ERL-1408: Cannot link OpenSSL library statically to compile crypto

[OTP-Maintainer]

Original reporter: JIRAUSER14904
Affected version: OTP-23.1.1
Component: crypto
Migrated from: https://bugs.erlang.org/browse/ERL-1408

---

Hello,

I am trying to compile OTP for Android (aarch64) with crypto library.

I have tried several things:
 * Link to openssl dynamic library compiled for android by using the option --with-ssl=. After running ./otp_build configure I get the following message at the end:

{{*********************************************************************}}
{{********************** APPLICATIONS DISABLED **********************}}
{{*********************************************************************}}

{{crypto : OpenSSL is configured for kerberos but no krb5.h found}}
{{odbc : ODBC library - link check failed}}
{{ssh : OpenSSL is configured for kerberos but no krb5.h found}}
{{ssl : OpenSSL is configured for kerberos but no krb5.h found}}
{{wx : User gave --without-wx option}}
 * {{Using the openssl library compiled for my host (linux). The configure passes without problem, however when running make I get the errors: skipping incompatible /usr/local/ssl/lib/libcrypto.so when searching for -lcrypto. Of course this makes sense, since it is using the clang compiler for android.}}

{{The openSSL is downloaded from the official repository, compiled for aarch64 using the specified instructions. Any help would be appreciated.}}

[OTP-Maintainer]

JIRAUSER14904 said:

Status update:

I recompiled the OpenSSL (openssl-1.1.1h) for android and on further inspection of Makefile located in crypto library, I noticed that by default the /lib/ path is prepended to the path specified, when the original compilation of OpenSSL saves all files in root location. After modifying the Makefile by hand with correct path the project compiled with success.

After deploy on android the application crashes with message:

args: [load_failed,"Failed to load NIF library /data/data/com.isotope11.erlanglauncher/files/test_phoenix_android/lib/crypto-4.8/priv/lib/crypto: 'dlopen failed: library \"libcrypto.so.1.1\" not found'","OpenSSL might not be installed on this system.\n"]

format: "Unable to load crypto library. Failed with error:~n\"~p, ~s\"~n~s"

label: \{error_logger,error_msg}

Is it possible to embed the libcrypto at compile time, or the only possibility is to link it at runtime?

[OTP-Maintainer]

sverker said:

OTP configure option {{--disable-dynamic-ssl-lib}} should direct crypto to link against static libcrypto.a. OpenSSL must have been built to generate static libs (don't know if it's done by default).

 

[OTP-Maintainer]

JIRAUSER14904 said:

I can confirm that passing --disable-dynamic-ssl-lib does not create the correct Makefile, the dynamic link option is still yes. Even when manually changing the option to no the final files require the libcrypto.so.1.1 library.

To prove this, here is the output of objdump -p crypto.so| grep .so:

crypto.so: file format elf64-little
 NEEDED libcrypto.so.1.1
 NEEDED libdl.so
 NEEDED libc.so
 required from libc.so:
 required from libcrypto.so.1.1:

[OTP-Maintainer]

sverker said:

Yes this is a mess.

 

I tried {{--disable-dynamic-ssl-lib}} on my Ubuntu and still got dynamic linking like you.

Then I added {{--with-ssl=/usr/lib/x86_64-linux-gnu}} where my libcrypto.a is located, but got the same result.

It seems {{--with-ssl}} wants a path it can append "/lib" and "/include" to, so I made a temporary directory where I created two soft links "lib" and "include" pointing to my /usr/lib/x86_64-linux-gnu and /usr/include/x86_64-linux-gnu and THEN IT WORKED. 

 

By the way, a nicer way to see which dynamic link dependencies you have is to use "ldd crypto.so"

 

[OTP-Maintainer]

JIRAUSER14904 said:

Just to make this clear, is libcrypto just dynamically linked or it is used to compile crypto in OTP?

Because it would make no sense to use a version compiled for linux. In documentation is stated:

OpenSSL -- The opensource toolkit for Secure Socket Layer and Transport Layer Security. Required for building the application crypto. Further, ssl and ssh require a working crypto application and will also be skipped if OpenSSL is missing. The public_key application is available without crypto, but the functionality will be very limited.

[OTP-Maintainer]

sverker said:

We don't test Android and I don't have access to it. I was just describing my attemts on Linux in hope it migt give you some hints of what to try.

 

libcrypto in OpenSSL can either by dynamically or statically linked to crypto.so in OTP.
 * If statically linked, file libcrypto.a must be found at compile time as its code will be baked into crypto.so.
 * If dynamically linked, a libcrypto.so file must also be found at compile time. I'm not sure why and what it actually uses it for as it may very well be a different (but compatible) libcrypto.so file that is actually dynamically linked at runtime.

 

[OTP-Maintainer]

JIRAUSER14904 said:

Thanks!

This does seem to make sense, I will check the configure and Makefiles maybe there is a bug somewhere.

[OTP-Maintainer]

ingela said:

Maybe the changes [https://github.com/erlang/otp/pull/2876]  here have some relevance for you.

[OTP-Maintainer]

jeromedebretagne said:

Hi @D4no0,

One assumption is that Android doesn't like libraries with version suffix, I've seen this point mentioned on the OpenSSL bug tracker, for instance here: https://github.com/openssl/openssl/issues/3902 

I've also been able to compile Erlang for Android using the -with-ssl option (I had to create a lib folder in my openssl local build for aarch64, with symlinks to the libraries too, that's another bug to raise separately) so I have a version linked dynamically locally. I didn't face the "no krb5.h found" error on my side though.

Could you share your test program that you've tried in Comment #1 or another minimal test case that lead to the "dlopen failed: library \"libcrypto.so.1.1\" not found" message? I'd be keen to investigate this issue on my side too with an actual Android device and see how to provide and expose the library so that it gets detected and loaded correctly. 

My first investigation steps would be to :
1/ link with libcrypto.so (instead of libcrypto.so.1.1)
2/ include libcrypto.so in your jniLibs/<abi> directory in your Android project
3/ see if it gets loaded automatically from this path 
4/ and if not, check the envionment option to pass in order to expand the directories used to load libraries from your native `erl` call.

You seem to be very close in fact!

Cheers, Jérôme

[OTP-Maintainer]

JIRAUSER14904 said:

"no krb5.h found" error is related to wrong path of the openssl, as mentioned above, it always searches the libcrypto.so in /lib folder, if you update the Makefile to point to root folder or move the libcrypto.so to /lib folder it works without problems.

Sadly you *cannot* load libraries with x.so.y format in Android, at least from JVM side, since the only way to load a dynamic library is to call {{System.loadLibrary(); and this function expects an argument in form "crypto", that then searches for a library called libcrypto.so.}}

I have tried to compile OpenSSL so that the output has no numbers, I was able to produce an output file without numbers, however when linked dynamically on project compile it is still libcrypto.so.1.1 (I have to try to clean the compilation folders, maybe it was using some old precompiled binaries). Looks like there are some headers that still get the version added.

Generally I would like to link the library statically, to avoid adding complexity to the runtime and code, since the openssl libraries won't get changed after the OTP was compiled. I will update my fork tomorrow with a minimal example you can run.

[OTP-Maintainer]

wojtekmach said:

I've run into a similar issue on macOS.

I'm on latest OTP master ({{508ccbf}}) and I see it's dynamically linked to libcrypto:

{code}
./otp_build autoconf && ./configure --disable-dynamic-ssl-lib --with-ssl=$(brew --prefix openssl) && make && otool -L lib/crypto/priv/lib/**/crypto.so

lib/crypto/priv/lib/x86_64-apple-darwin19.6.0/crypto.so:
	/usr/local/opt/openssl@1.1/lib/libcrypto.1.1.dylib (compatibility version 1.1.0, current version 1.1.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)
{code}

I'm getting similar result on both:

- macOS 10.15.7 x86_64 + openssl from homebrew
- macOS 11.0.1 arm64 + openssl from macports

I tried the workaround with temporary directory + symlinks to lib and include, but no dice.

Any guidance will be really appreciated!

[OTP-Maintainer]

wojtekmach said:

The workaround from Sverker worked after all, my mistake was symlinking all libcrypto.* files. If I just symlink libcrypto.a though it does work. (Which makes sense, there's no *.dylib to link.)

[OTP-Maintainer]

ingela said:

Question answered. We will not do anything more as Android is not an official supported platform that we have a test environment for.

[OTP-Maintainer]

JIRAUSER14904 said:

I managed to build the project with static library, the only bug I noticed is the fact that passing option --disable-dynamic-ssl-lib has no effect. Currently the solution to this problem is to edit Makefile in crypto folder manually and change dynamic linking to no. When I will have free time I will try to find what the culprit is and make a PR.

[OTP-Maintainer]

ingela said:

Thanks. PR  is welcome and will be accepted as long as it does not break any of the tested platforms.

[OTP-Maintainer]

jeromedebretagne said:

That's excellent news @D4no0 that you've managed to build and link statically with OpenSSL for Android!

I'm trying to setup a public repository of Erlang/OTP releases built for Android here: 
https://github.com/JeromeDeBretagne/erlang-otp-releases-for-android

If you can share which edit(s) you've had to make in the Makefile for crypto, I'll be happy to try the same locally and then update the published Erlang/OTP releases to include all the apps relying on OpenSSL.  

Thanks, Jérôme

[OTP-Maintainer]

JIRAUSER14904 said:

I finally found the solution to the problem, here you can see:

 
{code:java}
if test '!' -f "${SSL_LIBDIR}/lib${SSL_CRYPTO_LIBNAME}.a"; then

SSL_DYNAMIC_ONLY=yes
{code}
 

This is a line from the Makefile.in located in crypto library source location. It will automatically link at runtime if it cannot find libcrypto.a in openssl/lib directory. The current workaround is to copy the file libcrypto.a manually to lib folder of the openssl path specified (options --openssldir= and --prefix= do not seem to work, the compiled files still remain in root).

After that to ensure that library is not linked at runtime, after running configure, file /otp/lib/crypto/c_src/aarch64-unknown-linux-android/Makefile should have the following set:
{code:java}
DYNAMIC_CRYPTO_LIB=no{code}
 

[OTP-Maintainer]

JIRAUSER14904 said:

I will do a PR to update instructions to mention this thing in INSTALL-ANDROID.MD and how to correctly build openssl for android.

[OTP-Maintainer]

sverker said:

It seems wrong that it falls back on dynamic linking when it can't find the static library if the user has asked for --disable-dynamic-ssl-lib.

 

 

[OTP-Maintainer]

JIRAUSER14904 said:

Indeed, it seems like a hack that was left. I have a feeling that changing this into an error might break some builds.

[OTP-Maintainer]

jeromedebretagne said:

{quote}I will do a PR to update instructions to mention this thing in INSTALL-ANDROID.MD and how to correctly build openssl for android.
{quote}
I've proposed [https://github.com/erlang/otp/pull/2957] to update the Android instructions so that it is properly documented.