compile issue with openssl 3 backend on RHEL 8.5 machine

[totti0858]

I try to compile the softhsm2 current developer branch in RHEL 8.5 machine (openssl version 1.1.1k) with openssl 3.0 back-end. But I confront with a compile issue with below steps:

1. git clone xxx
2. sh autogen.sh
3. ./configure --with-crypto-backend=openssl --with-openssl=/path to the compiled openssl 3.0 library/
4. make

But there has issue when compile softhsm2-util:
Entering directory '/home/oye/SoftHSMv2/src/bin/util'
/bin/sh ../../../libtool --tag=CXX --mode=link g++ -g -O2 -Wall -Wextra -fvisibility=hidden -o softhsm2-util softhsm2-util.o ../common/findslot.o ../common/getpw.o ../common/library.o softhsm2-util-ossl.o ../../lib/crypto/OSSLComp.o -L/home/oye/Documents/lin64+glibc217+gcc12/usr/lib -lcrypto ../../lib/libsofthsm_convarch.la -ldl
libtool: link: g++ -g -O2 -Wall -Wextra -fvisibility=hidden -o softhsm2-util softhsm2-util.o ../common/findslot.o ../common/getpw.o ../common/library.o softhsm2-util-ossl.o ../../lib/crypto/OSSLComp.o -L/home/oye/Documents/lin64+glibc217+gcc12/usr/lib ../../lib/.libs/libsofthsm_convarch.a -lcrypto -ldl
../../lib/.libs/libsofthsm_convarch.a(OSSLEDPrivateKey.o): In function OSSLEDPrivateKey::setFromOSSL(evp_pkey_st const*)': /home/oye/SoftHSMv2/src/lib/crypto/OSSLEDPrivateKey.cpp:104: undefined reference to EVP_PKEY_get_id'
../../lib/.libs/libsofthsm_convarch.a(OSSLEDPublicKey.o): In function OSSLEDPublicKey::setFromOSSL(evp_pkey_st const*)': /home/oye/SoftHSMv2/src/lib/crypto/OSSLEDPublicKey.cpp:106: undefined reference to EVP_PKEY_get_id'
../../lib/.libs/libsofthsm_convarch.a(OSSLEVPHashAlgorithm.o): In function OSSLEVPHashAlgorithm::hashFinal(ByteString&)': /home/oye/SoftHSMv2/src/lib/crypto/OSSLEVPHashAlgorithm.cpp:113: undefined reference to EVP_MD_get_size'
../../lib/.libs/libsofthsm_convarch.a(OSSLEVPMacAlgorithm.o): In function OSSLEVPMacAlgorithm::signFinal(ByteString&)': /home/oye/SoftHSMv2/src/lib/crypto/OSSLEVPMacAlgorithm.cpp:113: undefined reference to EVP_MD_get_size'
../../lib/.libs/libsofthsm_convarch.a(OSSLEVPMacAlgorithm.o): In function OSSLEVPMacAlgorithm::verifyFinal(ByteString&)': /home/oye/SoftHSMv2/src/lib/crypto/OSSLEVPMacAlgorithm.cpp:203: undefined reference to EVP_MD_get_size'
softhsm2-util-ossl.o: In function crypto_malloc_eddsa(evp_pkey_st*)': /home/oye/SoftHSMv2/src/bin/util/softhsm2-util-ossl.cpp:918: undefined reference to EVP_PKEY_get_id'
softhsm2-util-ossl.o: In function crypto_import_key_pair(unsigned long, char*, char*, char*, char*, unsigned long, int)': /home/oye/SoftHSMv2/src/bin/util/softhsm2-util-ossl.cpp:154: undefined reference to EVP_PKEY_get_id'
../../lib/.libs/libsofthsm_convarch.a(OSSLAES.o): In function OSSLAES::wrapUnwrapKey(SymmetricKey const*, SymWrap::Type, ByteString const&, ByteString&, int) const': /home/oye/SoftHSMv2/src/lib/crypto/OSSLAES.cpp:180: undefined reference to EVP_CIPHER_CTX_get_block_size'
collect2: error: ld returned 1 exit status
make[4]: *** [Makefile:527: softhsm2-util] Error 1

How to fix this compile issue?

[totti0858]

the path should be -L/home/oye/Documents/lin64+glibc217+gcc12/usr/lib64
instead of -L/home/oye/Documents/lin64+glibc217+gcc12/usr/lib

After modify the openssl library path, I confront another issue:
/bin/sh ../../../libtool --tag=CXX --mode=link g++ -g -O2 -Wall -Wextra -fvisibility=hidden -o softhsm2-util softhsm2-util.o ../common/findslot.o ../common/getpw.o ../common/library.o softhsm2-util-ossl.o ../../lib/crypto/OSSLComp.o -L/home/oye/Documents/lin64+glibc217+gcc12/usr/lib -lcrypto ../../lib/libsofthsm_convarch.la -ldl
libtool: link: g++ -g -O2 -Wall -Wextra -fvisibility=hidden -o softhsm2-util softhsm2-util.o ../common/findslot.o ../common/getpw.o ../common/library.o softhsm2-util-ossl.o ../../lib/crypto/OSSLComp.o -L/home/oye/Documents/lin64+glibc217+gcc12/usr/lib ../../lib/.libs/libsofthsm_convarch.a -lcrypto -ldl
/usr/bin/ld: /home/oye/Documents/lin64+glibc217+gcc12/usr/lib/libcrypto.a(libcrypto-lib-aes-x86_64.o): unable to initialize decompress status for section .debug_line
/usr/bin/ld: /home/oye/Documents/lin64+glibc217+gcc12/usr/lib/libcrypto.a(libcrypto-lib-aes-x86_64.o): unable to initialize decompress status for section .debug_line
/usr/bin/ld: /home/oye/Documents/lin64+glibc217+gcc12/usr/lib/libcrypto.a(libcrypto-lib-a_object.o): unable to initialize decompress status for section .debug_info
/usr/bin/ld: /home/oye/Documents/lin64+glibc217+gcc12/usr/lib/libcrypto.a(libcrypto-lib-a_object.o): unable to initialize decompress status for section .debug_info
/home/oye/Documents/lin64+glibc217+gcc12/usr/lib/libcrypto.a: error adding symbols: File format not recognized
collect2: error: ld returned 1 exit status
make[4]: *** [Makefile:527: softhsm2-util] Error 1