Odd Ubuntu test failure with 7.1.0

[nacc]

Hi, I'm an Ubuntu developer and looking for some assistance in debugging an odd failure we're seeing, as noticed here: https://launchpad.net/ubuntu/+source/heimdal/7.1.0+dfsg-9. Sorry for the long issue report, but I'm hopefully providing all the information that is needed.

The builds are currently failing for heimdal in 17.04's -proposed release: https://launchpadlibrarian.net/302116850/buildlog_ubuntu-zesty-amd64.heimdal_7.1.0+dfsg-9_BUILDING.txt.gz, due to a testcase failure FAIL: test_rfc3961 at build-time.

The actual output is:

./lib/krb5/test_rfc3961 
/build/heimdal-Z6c0Pz/heimdal-7.1.0+dfsg/lib/krb5/.libs/test_rfc3961: relocation error: /build/heimdal-Z6c0Pz/heimdal-7.1.0+dfsg/lib/krb5/.libs/test_rfc3961: symbol hc_PKCS5_PBKDF2_HMAC, version HEIMDAL_CRYPTO_1.0 not defined in file libhcrypto.so.4 with link time reference

This seems to be due to:

ldd ./lib/krb5/.libs/test_rfc3961
	linux-vdso.so.1 =>  (0x00007ffeb89f7000)
	libeatmydata.so => /usr/lib/x86_64-linux-gnu/libeatmydata.so (0x00007fb9d7d46000)
	libwind.so.0 => /build/heimdal-Z6c0Pz/heimdal-7.1.0+dfsg/lib/wind/.libs/libwind.so.0 (0x00007fb9d7b1d000)
	libhcrypto.so.4 => /usr/lib/x86_64-linux-gnu/libhcrypto.so.4 (0x00007fb9d78e7000)
	libasn1.so.8 => /usr/lib/x86_64-linux-gnu/libasn1.so.8 (0x00007fb9d7644000)
	libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007fb9d7439000)
	libheimbase.so.1 => /usr/lib/x86_64-linux-gnu/libheimbase.so.1 (0x00007fb9d722a000)
	libroken.so.18 => /usr/lib/x86_64-linux-gnu/libroken.so.18 (0x00007fb9d7014000)
	libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007fb9d6ddc000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fb9d6bbe000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb9d67f7000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb9d65f1000)
	/lib64/ld-linux-x86-64.so.2 (0x000055e7b0f1b000)
	libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fb9d63d6000)

and you can see that the libhcrypto in-use is that from the system, rather than from the current build.
I see that lib/krb5/test_rfc3961 contains LD_LIBRARY_PATH=/build/heimdal-Z6c0Pz/heimdal-7.1.0+dfsg/lib/hx509/.libs:/build/heimdal-Z6c0Pz/heimdal-7.1.0+dfsg/lib/wind/.libs:/usr/lib/x86_64-linux-gnu:/build/heimdal-Z6c0Pz/heimdal-7.1.0+dfsg/lib/hcrypto/.libs:/build/heimdal-Z6c0Pz/heimdal-7.1.0+dfsg/lib/asn1/.libs:/build/heimdal-Z6c0Pz/heimdal-7.1.0+dfsg/lib/base/.libs:/build/heimdal-Z6c0Pz/heimdal-7.1.0+dfsg/lib/roken/.libs:/usr/lib/libeatmydata.

I believe this is written as-such because of lib/krb5/.libs/librfc3961.la:

dependency_libs=' /build/heimdal-Z6c0Pz/heimdal-7.1.0+dfsg/lib/hx509/libhx509.la /build/heimdal-Z6c0Pz/heimdal-7.1.0+dfsg/lib/wind/libwind.la -L/usr/lib /usr/lib/x86_64-linux-gnu/libsqlite3.la /build/heimdal-Z6c0Pz/heimdal-7.1.0+dfsg/lib/hcrypto/libhcrypto.la /build/heimdal-Z6c0Pz/heimdal-7.1.0+dfsg/lib/asn1/libasn1.la -lcom_err /build/heimdal-Z6c0Pz/heimdal-7.1.0+dfsg/lib/base/libheimbase.la /build/heimdal-Z6c0Pz/heimdal-7.1.0+dfsg/lib/roken/libroken.la -lcrypt -ldl -lresolv -lpthread'

where libsqlite3.la is listed before libhcrypto.la. If I do the following:

--- heimdal-7.1.0+dfsg.orig/lib/krb5/Makefile.in
+++ heimdal-7.1.0+dfsg/lib/krb5/Makefile.in
@@ -1143,13 +1143,13 @@ librfc3961_la_LIBADD = \
        $(top_builddir)/lib/ipc/libheim-ipcc.la \
        $(top_builddir)/lib/wind/libwind.la \
        $(LIB_pkinit) \
-       $(use_sqlite) \
        $(LIB_com_err) \
        $(LIB_hcrypto) \
        $(LIB_libintl) \
        $(LIBADD_roken) \
        $(PTHREAD_LIBADD) \
        $(LIB_door_create) \
+       $(use_sqlite) \
        $(LIB_dlopen)

 lib_LTLIBRARIES = libkrb5.la

the .la and test script both contain 'correct' (in terms of this issue) paths.

[vdukhovni]

Unfortunately, libtool makes winning this Quixotic fight impossible. Heimdal's test suite unavoidably runs into installed libraries through a twisted maze of .la files. The best solution is to do builds in containers, chroot jails, ... where Heimdal is not installed.

I tried to fix this issue, and gave up in disgust... We may look at this again some day, but it is too difficult to fix at this time.

[nacc]

@vdukhovni Thank you for the response! I did find a potential issue in libtool itself, which I'm still discussing on their ML. It seems like the LD_LIBRARY_PATH for the generated wrappers does not elide system include paths, but all other path generators do. And no one seems to really know why :)

[nacc]

@vdukhovni thanks again for your response. I'll close this issue for now. Unfortunately for Ubuntu builds it is hard to avoid as heimdal is now installed by default (via a dependency chain on the libs). So if you want an easy reproduction testbed, you have one :)