[lighttpd] -mod-openssl fails

lighttpd: 1.4.56-2
lighttpd-mod-openssl: 1.4.56-2
Maintainer: @flyn-org

libopenssl: 1.1.1h-1
Maintainer: @cotequeiroz

Environment: mvebu (Turris Omnia / armv7l / Cortex-A9)

Description:

# lighttpd -tt -f /etc/lighttpd/lighttpd.conf

(../src/plugin.c.195) dlopen() failed for: /usr/lib/lighttpd/mod_openssl.so Error relocating /usr/lib/lighttpd/mod_openssl.so: EC_KEY_new_by_curve_name: symbol not found
(../src/server.c.1225) loading plugins finally failed

given the output and a public search I suspect this being caused by libopenssl being compiled https://github.com/openwrt/openwrt/blob/master/package/libs/openssl/Makefile#L206

OPENSSL_OPTIONS += no-ec2m

Subsequent TLS connectivity with lighttpd is not possible (least not with openssl backend)

# opkg files lighttpd-mod-openssl

Package lighttpd-mod-openssl (1.4.56-2) is installed on root and has the following files:
/etc/lighttpd/conf.d/30-openssl.conf
/usr/lib/lighttpd/mod_openssl.so

[cotequeiroz]

The EC_KEY_new_by_curve_name symbol does not depend on ec2m support, and is present in libcrypto.
I've checked it with current snapshot libopenssl using dltest, which is part of the unixodbc-tools package.

# dltest /usr/lib/libcrypto.so.1.1 EC_KEY_new_by_curve_name
SUCCESS: Loaded /usr/lib/libcrypto.so.1.1
SUCCESS: Found EC_KEY_new_by_curve_name

It may be that mod_openssl.so is being loaded without libcrypto.so.1.1 being loaded also. Then, dlopen will not find that (or any libcrypto) symbol.

@cotequeiroz thank you for checking. run the same dltest with the same outcome.

[flyn-org]

I can reproduce this on x86_64 using lighttpd 1.4.56-2. I have the same result as @cotequeiroz, namely EC_KEY_new_by_curve_name is present in libcrypto.

This seems to allow lighttpd to run:

LD_PRELOAD=libcrypto.so.1.1:libssl.so.1.1 lighttpd -f /etc/lighttpd/lighttpd.conf

Something in the build seems to have generated an openssl module that does not load its requisite libraries. I think ldd confirms this as well:

# ldd /usr/lib/lighttpd/mod_openssl.so 
	ldd (0x7f5c62121000)
	libc.so => ldd (0x7f5c62121000)
Error relocating /usr/lib/lighttpd/mod_openssl.so: EC_KEY_new_by_curve_name: symbol not found
Error relocating /usr/lib/lighttpd/mod_openssl.so: SSL_CONF_CTX_set_flags: symbol not found
Error relocating /usr/lib/lighttpd/mod_openssl.so: SSL_CONF_CTX_finish: symbol not found
...

The other modules cannot find only lighttpd's symbols, which of course come from the program itself rather than a shared library.

The version bump to 1.4.56 started to use Meson, so I suspect something is wrong there. I am looking into this more now, but would be glad to take suggestions.

lighttpd-mod-wolfssl seems to suffer a similar issue #14122

lighttpd-mod-gnutls works with no issues on this node, did not test lighttpd-mod-mbedtls

[cotequeiroz]

It needs to be linked against libcrypto.so.1.1 and libssl.so.1.1 by adding -lcrypto -lssl to the link command. Indeed, the switch to meson may have altered that.

[flyn-org]

The linker command used by the build follows. I see libssl and libnettle, but not libcrypto. It is not yet clear if this is because of something wrong in the package Makefile, the lighttpd build process itself, or a difference between the musl and glibc dynamic loaders. @gstrauss, do you know?

[211/211] x86_64-openwrt-linux-musl-gcc  -o src/mod_openssl.so src/mod_openssl.so.p/mod_openssl.c.o -L/home/mike/Scratch/openwrt-sdk-x86-64_gcc-8.4.0_musl.Linux-x86_64/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/usr/lib -L/home/mike/Scratch/openwrt-sdk-x86-64_gcc-8.4.0_musl.Linux-x86_64/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/lib -Wl,--as-needed -Wl,--allow-shlib-undefined -shared -fPIC -Wl,--start-group -Wl,-soname,mod_openssl.so -znow -zrelro /home/mike/Scratch/openwrt-sdk-x86-64_gcc-8.4.0_musl.Linux-x86_64/staging_dir/target-x86_64_musl/usr/lib/libssl.so /home/mike/Scratch/openwrt-sdk-x86-64_gcc-8.4.0_musl.Linux-x86_64/staging_dir/target-x86_64_musl/usr/lib/libnettle.so -Wl,--end-group -Wl,-rpath,/home/mike/Scratch/openwrt-sdk-x86-64_gcc-8.4.0_musl.Linux-x86_64/staging_dir/target-x86_64_musl/usr/lib -Wl,-rpath-link,/home/mike/Scratch/openwrt-sdk-x86-64_gcc-8.4.0_musl.Linux-x86_64/staging_dir/target-x86_64_musl/usr/lib

[gstrauss]

@flyn-org: I don't see a problem. Please help me to understand what you think is incorrect. libssl.so depends on libcrypto.so

[Edit: just catching up to the comments above @flyn-org. I had not realized that this is a new bug report and will try to reproduce]

[gstrauss]

Maybe the module build is pulling in a static library for libssl.a and therefore does not have libcrypto.a? If the module links against libssl.so, then libssl.so has a dependency on libcrypto.so.

[gstrauss]

I think I see an issue in lighttpd/src/meson.build that overwrites a variable that is re-used between modules. It would work fine if only one TLS module is selected, but looks like it might cause problems when building all of the TLS modules. That's why the mod_openssl.so link posted by @flyn-org shows libssl and libnettle instead of libssl and libcrypto.

[gstrauss]

Yes, this appears to be an issue with lighttpd/src/meson.build, specifically my mistakes in updating it for lighttpd 1.4.56. It looks like lighttpd-mod-openssl, lighttpd-mod-wolfssl, and lighttpd-mod-mbedtls are currently broken due to incorrect linker command lines. lighttpd-mod-gnutls and lighttpd-mod-nss do not appear to have this issue. In my testing, I was building the TLS modules individually to test the crypto from each of the TLS libraries when the crypto portion was included by non-TLS modules. I overlooked and did not build all of the TLS modules together. Sorry.

I am testing the following and will post a PR in a few hours if it works. @n8v8R please tag me in lighttpd-related issues. I only found out about this issue when @flyn-org tagged me an hour ago.

diff --git a/src/meson.build b/src/meson.build
index 71e4ec90..2d6ee2df 100644
--- a/src/meson.build
+++ b/src/meson.build
@@ -432,15 +432,20 @@ if get_option('with_mysql')
 endif
 
 libssl = []
-libx509 = []
 libcrypto = []
+libsslcrypto = []
 libgnutls = []
+libmbedtls = []
+libmbedcrypto = []
+libmbedx509 = []
+libwolfssl = []
 if get_option('with_openssl')
        # manual search:
        # header: openssl/ssl.h
        # function: SSL_new (-lssl)
        # function: BIO_f_base64 (-lcrypto)
        libssl = [ dependency('libssl') ]
+       libsslcrypto = [ dependency('libcrypto') ]
        libcrypto = [ dependency('libcrypto') ]
        conf_data.set('HAVE_OPENSSL_SSL_H', true)
        conf_data.set('HAVE_LIBSSL', true)
@@ -449,7 +454,7 @@ if get_option('with_wolfssl')
        # manual search:
        # header: wolfssl/ssl.h
        # function: wolfSSL_Init (-lwolfssl)
-       libssl = [ dependency('wolfssl') ]
+       libwolfssl = [ dependency('wolfssl') ]
        libcrypto = [ dependency('wolfssl') ]
        conf_data.set('HAVE_WOLFSSL_SSL_H', true)
 endif
@@ -459,8 +464,9 @@ if get_option('with_mbedtls')
        # function: mbedtls_cipher_info_from_type (-lmbedtls)
        # function: mbedtls_x509_get_name (-lmbedx509)
        # function: mbedtls_base64_encode (-lmbedcrypto)
-       libssl = [ compiler.find_library('mbedtls') ]
-       libx509 = [ compiler.find_library('mbedx509') ]
+       libmbedtls = [ compiler.find_library('mbedtls') ]
+       libmbedx509 = [ compiler.find_library('mbedx509') ]
+       libmbedcrypto = [ compiler.find_library('mbedcrypto') ]
        libcrypto = [ compiler.find_library('mbedcrypto') ]
        conf_data.set('HAVE_LIBMBEDCRYPTO', true)
 endif
@@ -1070,19 +1076,19 @@ endif
 
 if get_option('with_openssl')
        modules += [
-               [ 'mod_openssl', [ 'mod_openssl.c' ], libssl + libcrypto ],
+               [ 'mod_openssl', [ 'mod_openssl.c' ], libssl + libsslcrypto ],
        ]
 endif
 
 if get_option('with_wolfssl')
        modules += [
-               [ 'mod_wolfssl', [ 'mod_wolfssl.c' ], libcrypto ],
+               [ 'mod_wolfssl', [ 'mod_wolfssl.c' ], libwolfssl ],
        ]
 endif
 
 if get_option('with_mbedtls')
        modules += [
-               [ 'mod_mbedtls', [ 'mod_mbedtls.c' ], libssl + libx509 + libcrypto ],
+               [ 'mod_mbedtls', [ 'mod_mbedtls.c' ], libmbedtls + libmbedx509 + libmbedcrypto ],
        ]
 endif
 

[flyn-org]

Thank you, @gstrauss! I suppose the build system builds all of the TLS modules. I generally only build one, but @n8v8R's comments prompted me to build wolfSSL support too. I can see why this went undetected until we bumped the version on the package.

[gstrauss]

I submitted PR #14128 to fix this issue.