Test fails on a Debian testing (Bookworm) distribution

[babelouest]

The library builds successfully on a Debian Bookworm, using the packages libmemcached-dev and libhiredis-dev, but the test fail here:

test/check_cache.c:60:F:core:test_cache_memcache:0: Assertion 'rc == 1' failed: rc == 0, 1 == 1
test/check_cache.c:60:F:core:test_cache_redis:0: Assertion 'rc == 1' failed: rc == 0, 1 == 1

The failing test is the following:

static void _test_basic_cache(oauth2_cache_t *c)
{
	bool rc = false;
	char *value = NULL;

	rc = oauth2_cache_set(_log, c, "piet", "klaas", 2);
	ck_assert_int_eq(rc, true);

[zandbelt]

that's strange: when I build, package and run tests myself on Debian Bookworm, it succeeds; can you compare with the Dockerfile that I use?

FROM debian:bookworm
MAINTAINER hans.zandbelt@zmartzone.eu

RUN apt-get update && apt-get install -y apt-utils wget

RUN echo "Acquire::http::Timeout \"90\";" > /etc/apt/apt.conf.d/99timeout

RUN apt-get update && apt-get install -y build-essential devscripts dh-make pkg-config util-linux
RUN apt-get update && apt-get install -y gcc tar autoconf automake libtool

ENV DEB_BUILD_OPTIONS noddebs

ENV NGINX_VERSION 1.18.0
ENV OAUTH2_VERSION 1.4.4.1

ENV OAUTH2_TGZ liboauth2-${OAUTH2_VERSION}.tar.gz
ENV OAUTH2_ORIG_TGZ liboauth2-${OAUTH2_VERSION}.orig.tar.gz

RUN apt-get update && apt-get install -y libssl-dev libjansson-dev libcurl4-openssl-dev
RUN apt-get update && apt-get install -y libmemcached-dev memcached
RUN apt-get update && apt-get install -y libhiredis-dev redis-server
RUN apt-get update && apt-get install -y apache2-dev vim check

WORKDIR /root

RUN apt-get update && apt-get install -y libghc-regex-pcre-dev zlib1g-dev libxml2-dev libxslt-dev libgd-dev libgeoip-dev
RUN wget https://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz && tar zxvf nginx-${NGINX_VERSION}.tar.gz
RUN ln -s nginx-${NGINX_VERSION} nginx

RUN apt-get install -y libcjose-dev

COPY ${OAUTH2_TGZ} ${OAUTH2_ORIG_TGZ}
RUN tar zxvf ${OAUTH2_ORIG_TGZ}

WORKDIR /root/liboauth2-${OAUTH2_VERSION}
ADD debian-liboauth2 debian
RUN sed -i "s/bind .*/bind 127.0.0.1/g" /etc/redis/redis.conf

ENV CK_FORK no

RUN cd /root/nginx && ./configure --with-cc-opt='-g -O2 -ffile-prefix-map=/build/nginx-q9LD4J/nginx-${NGINX_VERSION}=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-compat --with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_sub_module

RUN service memcached start && service redis-server start && sleep 1 && dpkg-buildpackage -uc -us

and the results:

#26 11.27 make[1]: Leaving directory '/root/liboauth2-1.4.4.1'
#26 11.28 dh_auto_test
#26 11.35 make -j3 test VERBOSE=1
#26 11.35 make[1]: Entering directory '/root/liboauth2-1.4.4.1'
#26 11.35 make check_liboauth2
#26 11.36 make[2]: Entering directory '/root/liboauth2-1.4.4.1'
#26 11.36 gcc -DPACKAGE_NAME="liboauth2" -DPACKAGE_TARNAME="liboauth2" -DPACKAGE_VERSION="1.4.4.1" -DPACKAGE_STRING="liboauth2\ 1.4.4.1" -DPACKAGE_BUGREPORT="hans.zandbelt@zmartzone.eu" -DPACKAGE_URL="" -DHAVE_STDIO_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_STRINGS_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_UNISTD_H=1 -DSTDC_HEADERS=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=".libs/" -I. -Wall -Werror -Wno-error=deprecated-declarations -I./include -I./src -DHAVE_LIBMEMCACHE -DHAVE_LIBHIREDIS -Wall -Werror -Wno-error=deprecated-declarations -I./include -I./src -DHAVE_APACHE -Wall -Werror -Wno-error=deprecated-declarations -I./include -I./src -DHAVE_NGINX -Wdate-time -D_FORTIFY_SOURCE=2 -I/usr/include/x86_64-linux-gnu -pthread -pipe -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -DLINUX -D_REENTRANT -D_GNU_SOURCE -I/usr/include/apache2 -DLINUX -D_REENTRANT -D_GNU_SOURCE -I/usr/include/apr-1.0 -I /root/nginx/src/core -I /root/nginx/src/event -I /root/nginx/src/event/modules -I /root/nginx/src/os/unix -I /root/nginx/objs -I /root/nginx/src/http -I /root/nginx/src/http/v2 -I /root/nginx/src/http/modules -I/usr/include/apache2 -fPIC -c -o test/check_liboauth2-check_liboauth2.o test -f 'test/check_liboauth2.c' || echo './'test/check_liboauth2.c

...

#26 12.51 libtool: link: gcc -I/usr/include/x86_64-linux-gnu -pipe -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -DLINUX -D_REENTRANT -D_GNU_SOURCE -I/usr/include/apache2 -DLINUX -D_REENTRANT -D_GNU_SOURCE -I/usr/include/apr-1.0 -I /root/nginx/src/core -I /root/nginx/src/event -I /root/nginx/src/event/modules -I /root/nginx/src/os/unix -I /root/nginx/objs -I /root/nginx/src/http -I /root/nginx/src/http/v2 -I /root/nginx/src/http/modules -I/usr/include/apache2 -fPIC --coverage -o .libs/check_liboauth2 test/check_liboauth2-check_liboauth2.o test/check_liboauth2-check_version.o test/check_liboauth2-check_mem.o test/check_liboauth2-check_log.o test/check_liboauth2-check_cfg.o test/check_liboauth2-check_util.o test/check_liboauth2-check_ipc.o test/check_liboauth2-check_cache.o test/check_liboauth2-check_jose.o test/check_liboauth2-check_http.o test/check_liboauth2-check_proto.o test/check_liboauth2-check_oauth2.o test/check_liboauth2-check_openidc.o test/check_liboauth2-server_stubs.o test/check_liboauth2-check_apache.o test/check_liboauth2-check_nginx.o ./.libs/liboauth2_apache.so ./.libs/liboauth2.so /usr/lib/x86_64-linux-gnu/libapr-1.so /usr/lib/x86_64-linux-gnu/libaprutil-1.so -lldap -llber ./.libs/liboauth2_nginx.so -lssl -lcurl -lcjose -ljansson -lcrypto -lcheck_pic -lrt -lm -lsubunit -pthread
#26 12.63 make[2]: Leaving directory '/root/liboauth2-1.4.4.1'
#26 12.63 make check-TESTS
#26 12.64 make[2]: Entering directory '/root/liboauth2-1.4.4.1'
#26 12.65 make[3]: Entering directory '/root/liboauth2-1.4.4.1'
#26 32.85 PASS: check_liboauth2
#26 32.88 ============================================================================
#26 32.88 Testsuite summary for liboauth2 1.4.4.1
#26 32.88 ============================================================================
#26 32.88 # TOTAL: 1
#26 32.88 # PASS: 1
#26 32.88 # SKIP: 0
#26 32.88 # XFAIL: 0
#26 32.88 # FAIL: 0
#26 32.88 # XPASS: 0
#26 32.88 # ERROR: 0
#26 32.88 ============================================================================
#26 32.88 make[3]: Leaving directory '/root/liboauth2-1.4.4.1'

[babelouest]

Actually, I believe the problem comes from me, in my Debian Bookworm docker instance, I install libmemcached-dev and libhiredis-dev, but no redis-server or memchached, if I install them and run the services (as in your Docker file), the test succeeds. My bad!

In your opinion, would it be possible to mock the redis and the memcached servers during the tests?

[zandbelt]

In your opinion, would it be possible to mock the redis and the memcached servers during the tests?

I have the feeling I'd be debugging the mock more than testing the actual code.
How about skipping those tests using an environment variable setting?

[babelouest]

How about skipping those tests using an environment variable setting?

Sort of, I patched the check_cache.c to disable redis and memchache tests, now it works on my side.

Although the package still fails to build on Debian servers: https://buildd.debian.org/status/fetch.php?pkg=liboauth2&arch=amd64&ver=1.4.4.1-2&stamp=1650030750&raw=0

The first test to fail is the http_server_spawn:

Running suite(s): liboauth2
 version
[test/check_liboauth2.c:271:http_server_spawn:ERR] bind failed

If I force this test to fail on my machine, I can reproduce the failing list. So I assume that if I can make the test http server running, the package will build successfully on Debian servers. Now the goal is to find what's wrong. Is it the port number (8888), is it the bind options? I'll make some tests and I'll let you know.

[babelouest]

Hello @zandbelt , I'm getting closer!

I have changed the http_server_spawn port number to 8080 and now almost all architectures are passing the tests on Debian buildd!

I have one new issue with the armhf arch which fails at a new place though:

test/check_cache.c:87:F:core:test_cache_file:0: Assertion 'rc == 1' failed: rc == 0, 1 == 1

The test fails at the following line:

rc = oauth2_cache_set(_log, c, "piet", NULL, 0);
ck_assert_int_eq(rc, true);

Note: I don't think it's an architecture issue, because I have the same problem on the Salsa CI which uses amd64 architecture: https://salsa.debian.org/debian-iot-team/oauth2/liboauth2/-/jobs/2676634

[zandbelt]

I see the following in the log:

[src/cache/file.c:479:oauth2_cache_file_set:DBG] enter
[src/cache/file.c:379:_oauth2_cache_files_clean:DBG] start cleaning cycle
[src/cache/file.c:431:_oauth2_cache_files_clean:DBG] cache entry (oauth2-cache-piet) expired, removing file "/tmp/oauth2-cache-piet")
[src/cache/file.c:224:_oauth2_cache_file_remove:ERR] could not delete cache file "/tmp/oauth2-cache-piet" (No such file or directory)
[src/cache/file.c:535:oauth2_cache_file_set:DBG] leave: 0
[src/cache.c:431:oauth2_cache_set:ERR] leave: could NOT store: piet

and I'm pretty sure it is a timing issue or race condition similar to the one I fixed only recently in mod_auth_openidc; I will look in to it soon and report back

[zandbelt]

can you try https://github.com/zmartzone/liboauth2/releases/tag/v1.4.4.2 ?

[babelouest]

Looks like it works!

I'll upload to unstable in the next days, thanks!