Reg-tests timeouts

[Tristan971]

Tool Name and Version

N/A

Code Report

Before:

VTEST_PROGRAM="/builds/mangadex-pub/haproxy/deps/vtest/dist/vtest" make -C "src" -j "8" REGTESTS_TYPES=default,bug,devel reg-tests
...
########################## Starting vtest ##########################
Testing with haproxy version: 3.0-dev4-7151076+mangadex-73c78ef
Warning: version does not match the current tree (3.0-dev4-7151076)
0 tests failed, 20 tests skipped, 151 tests passed

-------

After:

VTEST_PROGRAM="/builds/mangadex-pub/haproxy/deps/vtest/dist/vtest" make -C "src" -j "8" REGTESTS_TYPES=default,bug,devel reg-tests
...
########################## Starting vtest ##########################
Testing with haproxy version: 3.0-dev8-e158b7e+mangadex-63ba2a6
Warning: version does not match the current tree (3.0-dev8-e158b7e)
#    top  TEST reg-tests/http-rules/acl_cli_spaces.vtc FAILED (2.194) exit=2
#    top  TEST reg-tests/mcli/mcli_show_info.vtc FAILED (2.186) exit=2
#    top  TEST reg-tests/mcli/mcli_start_progs.vtc FAILED (2.188) exit=2
#    top  TEST reg-tests/connection/http_reuse_always.vtc FAILED (2.182) exit=2
4 tests failed, 20 tests skipped, 149 tests passed
########################## Gathering results ##########################
###### Test case: reg-tests/mcli/mcli_start_progs.vtc ######
## test results in: "/tmp/haregtests-2024-04-24_21-15-27.ZNxRWp/vtc.1745.7df4f89b"
---- h1    Bad exit status: 0x0000 exit 0x0 signal 0 core 0
---- h1    Assert error in haproxy_wait(), src/vtc_haproxy.c line 811:  Condition(*(&h->fds[1]) >= 0) not true.  Errno=0 Success
*    diag  0.0 /usr/bin/sleep
###### Test case: reg-tests/mcli/mcli_show_info.vtc ######
## test results in: "/tmp/haregtests-2024-04-24_21-15-27.ZNxRWp/vtc.1745.2fa909c3"
---- h1    Bad exit status: 0x0000 exit 0x0 signal 0 core 0
---- h1    Assert error in haproxy_wait(), src/vtc_haproxy.c line 811:  Condition(*(&h->fds[1]) >= 0) not true.  Errno=0 Success
###### Test case: reg-tests/http-rules/acl_cli_spaces.vtc ######
## test results in: "/tmp/haregtests-2024-04-24_21-15-27.ZNxRWp/vtc.1745.5deb098a"
---- h1    Bad exit status: 0x0000 exit 0x0 signal 0 core 0
###### Test case: reg-tests/connection/http_reuse_always.vtc ######
## test results in: "/tmp/haregtests-2024-04-24_21-15-27.ZNxRWp/vtc.1745.6b97a59a"
---- h1    Bad exit status: 0x0000 exit 0x0 signal 0 core 0
make[1]: *** [Makefile:1206: reg-tests] Error 1

Additional Information

As I was bisecting for #2561, I noticed that my builds took much longer to finish (an extra 5-6 minutes).

After having a look, it seems that the cause is a few regtests regularly failing with timeouts (see above), which was not the case in 7151076 but is the case in e158b7e.

It is suspicious as my build environment/options are not very original. I'll investigate after my bisecting is done, but this issue will do as a reminder.

for reference:

• before
• after

Output of haproxy -vv

HAProxy version 3.0-dev8-e158b7e+mangadex-63ba2a6 2024-04-24T21:13+00:00 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
Running on: Linux 5.15.0-101-generic #111~20.04.1-Ubuntu SMP Mon Mar 11 15:44:43 UTC 2024 x86_64
Build options :
  TARGET  = linux-glibc
  CC      = cc
  CFLAGS  = -O2 -g -fwrapv -DMAX_SESS_STKCTR=5
  OPTIONS = USE_LIBCRYPT=1 USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_TFO=1 USE_NS=1 USE_SYSTEMD=1 USE_QUIC=1 USE_PROMEX=1 USE_STATIC_PCRE2=1 USE_PCRE2=1 USE_PCRE2_JIT=1
  DEBUG   = -DDEBUG_MEMORY_POOLS -DDEBUG_STRICT
Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE +STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB
Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=8).
Built with OpenSSL version : OpenSSL 1.1.1w+quic-mangadex-63ba2a6 24 Apr 2024
Running on OpenSSL version : OpenSSL 1.1.1w+quic-mangadex-63ba2a6 24 Apr 2024
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.4.6
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.42 2022-12-11
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with clang compiler version 17.0.6 (++20231208085813+6009708b4367-1~exp1~20231208085906.81)
Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
       quic : mode=HTTP  side=FE     mux=QUIC  flags=HTX|NO_UPG|FRAMED
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
Available services : prometheus-exporter
Available filters :
	[BWLIM] bwlim-in
	[BWLIM] bwlim-out
	[CACHE] cache
	[COMP] compression
	[FCGI] fcgi-app
	[SPOE] spoe
	[TRACE] trace
ldd /builds/mangadex-pub/haproxy/haproxy/dist/usr/local/sbin/haproxy
	linux-vdso.so.1 (0x00007ffc272e0000)
	libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007529f190b000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007529f182c000)
	libatomic.so.1 => /lib/x86_64-linux-gnu/libatomic.so.1 (0x00007529f1822000)
	libjemalloc.so.2 => /lib/x86_64-linux-gnu/libjemalloc.so.2 (0x00007529f1535000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007529f1354000)
	/lib64/ld-linux-x86-64.so.2 (0x00007529f237c000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007529f1138000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007529f1118000)

[wtarreau]

Please note that there have been chaotic versions which could possibly not leave immediately, so it's not completely impossible that you've faced one of them. I'd say that if this continues to happen with latest -dev then it's a real concern, otherwise it's perfectly possible that it has since been fixed.

[Tristan971]

I'd say that if this continues to happen with latest -dev then it's a real concern

It does happen at least up to 03ca16f here (see https://gitlab.com/mangadex-pub/haproxy/-/jobs/6785851604).

########################## Starting vtest ##########################
Testing with haproxy version: 3.0-dev10-03ca16f+mangadex-2e432d6
Warning: version does not match the current tree (3.0-dev10-03ca16f)
#    top  TEST reg-tests/http-rules/acl_cli_spaces.vtc FAILED (2.206) exit=2
#    top  TEST reg-tests/mcli/mcli_show_info.vtc FAILED (2.189) exit=2
#    top  TEST reg-tests/mcli/mcli_start_progs.vtc FAILED (2.181) exit=2
#    top  TEST reg-tests/connection/http_reuse_always.vtc FAILED (2.174) exit=2
4 tests failed, 20 tests skipped, 150 tests passed
########################## Gathering results ##########################
###### Test case: reg-tests/connection/http_reuse_always.vtc ######
## test results in: "/tmp/haregtests-2024-05-06_09-54-03.z9bbAH/vtc.1749.30c6b44a"
---- h1    Bad exit status: 0x0000 exit 0x0 signal 0 core 0
###### Test case: reg-tests/mcli/mcli_show_info.vtc ######
## test results in: "/tmp/haregtests-2024-05-06_09-54-03.z9bbAH/vtc.1749.3395aaba"
---- h1    Bad exit status: 0x0000 exit 0x0 signal 0 core 0
---- h1    Assert error in haproxy_wait(), src/vtc_haproxy.c line 811:  Condition(*(&h->fds[1]) >= 0) not true.  Errno=0 Success
###### Test case: reg-tests/http-rules/acl_cli_spaces.vtc ######
## test results in: "/tmp/haregtests-2024-05-06_09-54-03.z9bbAH/vtc.1749.32bbbf95"
---- h1    Bad exit status: 0x0000 exit 0x0 signal 0 core 0
###### Test case: reg-tests/mcli/mcli_start_progs.vtc ######
## test results in: "/tmp/haregtests-2024-05-06_09-54-03.z9bbAH/vtc.1749.2c8328c3"
---- h1    Bad exit status: 0x0000 exit 0x0 signal 0 core 0
---- h1    Assert error in haproxy_wait(), src/vtc_haproxy.c line 811:  Condition(*(&h->fds[1]) >= 0) not true.  Errno=0 Success
*    diag  0.0 /usr/bin/sleep

I just reported the first "faulty" (if the issue is not fully on my side) commit I found in my build history.

That said, looking at this repo's GH Actions doesn't seem to show the same issue, so I'm not sure yet if it's legitimate.

[Tristan971]

Interestingly, it doesn't happen in my GH Actions either (see: https://github.com/mangadex-pub/haproxy/actions/runs/8989710831/job/24693449238), so it is most likely dependency on some system permissions for something in those tests.

Permissions which we presumably don't grant to containers in our k8s cluster running Gitlab CI pipelines.