Feature request: make varnishtest -i discover libvarnishapi in the build directories

[ingvarha]

Note that the following problem is specific for fedora/redhat rpm builds. I have no idea why this problem does not show up when doing a plain ./configure && make && make check

It would be nice to run the varnishtest check suite within a build without using LD_LIBRARY_PATH. varnishtest -i will discover most binaries and libraries from the build directories, but does not find libvarnishapi. Detailed examples below.

Ingvar

After an initial fedora rpm build*, I may call varnishtest from within the build tree like this:

[ingvar@netcat varnishtest (master)]$ ./varnishtest -i
/home/ingvar/git/fedora/varnish/varnish-6.0.1/bin/varnishtest/.libs/lt-varnishtest: error while loading shared libraries: libvarnishapi.so.1: cannot open shared object file: No such file or directory

[ingvar@netcat varnishtest (master)]$ LD_LIBRARY_PATH=~/git/fedora/varnish/varnish-6.0.1/lib/libvarnishapi/.libs ./varnishtest -i
usage: varnishtest [options] file ...
-b size # Set internal buffer size (default: 1M)
-D name=val # Define macro
-i # Find varnish binaries in build tree
-j jobs # Run this many tests in parallel
-k # Continue on test failure
-L # Always leave temporary vtc.*
-l # Leave temporary vtc.* if test fails
-n iterations # Run tests this many times
-p name=val # Pass a varnishd parameter
-q # Quiet mode: report only failures
-t duration # Time tests out after this long
-v # Verbose mode: always report test log
-W # Enable the witness facility for locking

[ingvar@netcat varnishtest (master)]$ make check 2>&1 | head
make check-TESTS check-local
make[1]: Entering directory '/home/ingvar/git/fedora/varnish/varnish-6.0.1/bin/varnishtest'
make[2]: Entering directory '/home/ingvar/git/fedora/varnish/varnish-6.0.1/bin/varnishtest'
FAIL: tests/a00000.vtc
FAIL: tests/a00001.vtc
FAIL: tests/a00002.vtc
FAIL: tests/a00003.vtc
FAIL: tests/a00004.vtc
FAIL: tests/a00005.vtc
FAIL: tests/a00006.vtc

[ingvar@netcat varnishtest (master)]$ LD_LIBRARY_PATH=~/git/fedora/varnish/varnish-6.0.1/lib/libvarnishapi/.libs make check | head
make check-TESTS check-local
make[1]: Entering directory '/home/ingvar/git/fedora/varnish/varnish-6.0.1/bin/varnishtest'
make[2]: Entering directory '/home/ingvar/git/fedora/varnish/varnish-6.0.1/bin/varnishtest'
PASS: tests/a00000.vtc
SKIP: tests/a00001.vtc
PASS: tests/a00002.vtc
PASS: tests/a00003.vtc
PASS: tests/a00004.vtc
PASS: tests/a00005.vtc
PASS: tests/a00006.vtc

*) The fedora build on x86_64 is called like this:

CFLAGS='-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection'

LDFLAGS='-Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld'

./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --disable-static --localstatedir=/var/lib --docdir=/usr/share/doc/varnish

[dridi]

After an initial fedora rpm build*, I may call varnishtest from within the build tree like this:

[ingvar@netcat varnishtest (master)]$ ./varnishtest -i
/home/ingvar/git/fedora/varnish/varnish-6.0.1/bin/varnishtest/.libs/lt-varnishtest: error while loading shared libraries: libvarnishapi.so.1: cannot open shared object file: No such file or directory

Before you run this command, what is the output of file ./varnishtest?

[nigoroll]

IRC disucssion: It seems to be that for the fedora builds, the libtool-generated wrapper scripts lack the LD_LIBRARY_PATH to find libvarnishapi from the build tree. This is the problem which should be addressed, IMHO.

[ingvarha]

varnishtest: POSIX shell script, ASCII text executable, with very long lines

[ingvarha]

IRC disucssion: It seems to be that for the fedora builds, the libtool-generated wrapper scripts lack the LD_LIBRARY_PATH to find libvarnishapi from the build tree. This is the problem which should be addressed, IMHO.

Yep. And the reason behind this is Fedora's rules against the use of rpath: https://docs.fedoraproject.org/en-US/packaging-guidelines/#_beware_of_rpath

As described in the guidelines, I do

sed -i 's|^hardcode_libdir_flag_spec=.*|hardcode_libdir_flag_spec=""|g' libtool
sed -i 's|^runpath_var=LD_RUN_PATH|runpath_var=DIE_RPATH_DIE|g' libtool

And as I do make check before installing the binaries, varnishtest is unable to locate its necessary libraries (only libvarnishapi, iirc).

[ingvarha]

I reached out for some help on the fedora-devel ml. Looks like libtool supports some magic to handle this without patching by overriding LT_SYS_LIBRARY_PATH=path or LT_SYS_LIBRARY_PATH=path: (both seem to work for me)

./configure LT_SYS_LIBRARY_PATH=%_libdir

On fedora & friends, %_libdir is /usr/lib or /usr/lib64

I'm testing builds now.

Ingvar

[ingvarha]

... and with a little luck, in the future, fedora/redhat may do this automatically for us :-)

https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/55

[ingvarha]

This change seem to work fine in my builds for fedora, el6, and el7, so I'll just close this ticket.