3.3.10: Stick-Table segfaults due to ACLs?

[idl0r]

Detailed Description of the Problem

Hi,

since the upgrade to 3.3.10 we got some stick-table related segfaults.
I don't recall but we had 3.3.9 and .8 as well for a very short time IIRC so not sure if the issue was already present in one of those. It's the first time with 3.3.10 however but we got at least two segfaults. I haven't checked the second one yet but I guess it is the same / related.

Expected Behavior

No crash

Steps to Reproduce the Behavior

N/A

Do you have any idea what may have caused this?

No response

Do you have an idea how to solve the issue?

No response

What is your configuration?

N/A

Output of haproxy -vv

HAProxy version 3.3.10-0a72d7f9f 2026/05/11 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2027.
Known bugs: http://www.haproxy.org/bugs/bugs-3.3.10.html
Running on: Linux 6.1.0-48-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.172-1 (2026-05-15) x86_64
Build options : 
  TARGET  = linux-glibc
  CC      = cc
  CFLAGS  = -O2 -g -march=x86-64-v3 -mtune=znver1 -fwrapv -fvect-cost-model=very-cheap
  OPTIONS = USE_LIBCRYPT=1 USE_OPENSSL_AWSLC=1 USE_LUA=1 USE_ZLIB= USE_SLZ=1 USE_NS= USE_PROMEX=1 USE_PCRE= USE_PCRE_JIT= USE_PCRE2=1 USE_PCRE2_JIT=
  DEBUG   = -DDEBUG_GLITCHES=1

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ECH -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE +KTLS -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER -NS -OBSOLETE_LINKER +OPENSSL +OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 -PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION -QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB +ACME +HAVE_TCP_MD5SIG

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=32, MAX_THREADS=1024, default=128).
Built with SSL library version : OpenSSL 1.1.1 (compatible; AWS-LC 1.73.0)
Running on SSL library version : AWS-LC 1.73.0
SSL library supports TLS extensions : yes
SSL library supports SNI : yes
SSL library FIPS mode : no
SSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.6
Built with the Prometheus exporter as a service
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.42 2022-12-11
PCRE2 library supports JIT : no (USE_PCRE2_JIT not set)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 12.2.0

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
       spop : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
	[BWLIM] bwlim-in
	[BWLIM] bwlim-out
	[CACHE] cache
	[COMP] compression
	[FCGI] fcgi-app
	[SPOE] spoe
	[TRACE] trace

Last Outputs and Backtraces

Coredumps available. Let me know if you need those.
Log: https://gist.github.com/idl0r/c5bbd510686c09432f854dc7efb16253

Backtrace: https://gist.github.com/idl0r/09b863e287d891173fba2b9f726706b5

Additional Information

No response

[idl0r]

The log of the second one. Seems to be something else, right? https://gist.github.com/idl0r/10203385280917588426b226e2107dd9
I'd open a different issue then.

[idl0r]

Hm, looking more into it, it might be the same.

[wtarreau]

Thanks Christian.

So there are multiple patterns here:

• t35,60,86 are in stktable_set_entry() via peer_treat_updatemsg(), waiting on the readlock(bucket.sh_lock)
• t28 is in stktable_add_pend_updates() on a trylock(updt_lock)
• t14, t43 and t68 are waiting in peer_send_teachmsgs() on a readlock(updt_lock)
• t1,10,25,26 are in process_tables_expire(), waiting on the wrlock(bucket.sh_lock)
• almost ll other ones are in stktable_get_entry() waiting on readlock(bucket.sh_lock)

This smells like lock inversion, we need to figure if any thread possibly holds the lock other ones are waiting for. I'm a bit surprised to see the trylock() being interrupted by the watchdog. It would have to be an amazing coincidence that it stops there, and I'm wondering if in fact it's not part of the problem.

[idl0r]

FTR: It didn't crash again since I reported those two crashes.

[wtarreau]

Thanks for letting me know. I'm wondering if it could be a rare case of extreme contention. There are so many read lockers that maybe the write locker has difficulties finding its way through this. With that said, write has precedence and readers may only read and not perform any cache line write once they find a writer, so once they're waiting, they should quickly let the writer finish. Or maybe that's another case of too many operations being done in a row.

I'm noting that there are updates coming in quantity form 3 peers while we're sending to 3 peers at the same time.

BTW I have not identified the thread holding the sh_lock, to be sure it's not waiting in turn...

[wtarreau]

Extremely strange, it's said in the panic dump that thread 25 (=0x7f94117296c0 = t13 in gdb) holds the table lock for writes, though it's in an allocation, totally outside of stick-table stuff. I hope we won't discover a path that could leak a lock :-/

[idl0r]

Let me know if you need the second backtrace and/or the dumps etc.

[wtarreau]

Might possibly help, at least to see if the pattern looks similar.

[wtarreau]

Thanks for the elements. In both case we're seeing process_tables_expire() in its first iteration in different threads, but it does. Once we've seen a slow pool_free() releasing memory to the system being called under this lock, and it's likely that it happens frequently during this purge. If this happens, other threads waiting for the update lock on the table will not be able to make progress. We have ideas about how to move objects to a temporary list and free them out of the lock to shorten the lock holding duration. This lock remains one of those with the most contention on tables because it cannot be sharded due to the need for compatibility with older versions of the protocol (i.e. some ordering guarantees are mandated between updates). But it's possible that freeing such objects out of the lock does the trick.

[wtarreau]

Olivier pushed a change to release objects outside of the lock. It will definitely improve the worst case and may even noticeably improve the average case because the lock will be held for a much shorter time around this loop. I'm going to issue dev13 in a few minutes, maybe you'd want to give it a try (if everything goes well, it should be the final -dev).

[idl0r]

Yeah, I'll deploy dev13 on our test LBs. I don't think we'll notice anything tho. It didn't crash again yet so it seems to have been a rather rare event.