Opentracing filter doesn't work with chroot in the HAproxy 2.4

[Kaminari-Y]

Detailed description of the problem

We are trying to PoC the opentracing implementation available in the HAproxy 2.4, everything works well unless we add the chroot to the config file.
When we add the chroot and restart the service, it fails with the below error.

May 31 14:25:48 host haproxy[28778]: [NOTICE]   (28784) : haproxy version is 2.4.0-6cbbecf
May 31 14:25:48 host haproxy[28778]: [NOTICE]   (28784) : path to executable is /usr/sbin/haproxy
May 31 14:25:48 host haproxy[28778]: [ALERT]    (28784) : ‘filter’ : ‘opentracing’ : ‘/etc/haproxy/ctx/cfg-jaeger.yml’:
May 31 14:25:48 host haproxy[28778]: [ALERT]    (28784) : Failed to initialize filters for proxy ‘ray-mh-port-80’ for thread 1.
May 31 14:25:48 host haproxy[28778]: [ALERT]    (28784) : failed to initialize thread 1.
May 31 14:25:48 host haproxy[28778]: [WARNING]  (28778) : Current worker #1 (28784) exited with code 1 (Exit)
May 31 14:25:48 host haproxy[28778]: [NOTICE]   (28778) : haproxy version is 2.4.0-6cbbecf
May 31 14:25:48 host haproxy[28778]: [NOTICE]   (28778) : path to executable is /usr/sbin/haproxy
May 31 14:25:48 host haproxy[28778]: [ALERT]    (28778) : exit-on-failure: killing every processes with SIGTERM
May 31 14:25:48 host haproxy[28778]: [WARNING]  (28778) : All workers exited. Exiting... (1)

Our configuration is -

global
  user haproxy
  group haproxy
  nbproc 1
  nbthread 16
  cpu-map auto:1/1-16 0-15
  log /dev/log local2
  log /dev/log local0 notice
  chroot /var/lib/haproxy
  pidfile /var/run/haproxy.pid
  daemon
  master-worker
  maxconn 200000
  hard-stop-after 1h
  stats socket /var/lib/haproxy/stats mode 660 level admin expose-fd listeners
  tune.ssl.cachesize $size
  tune.ssl.lifetime $size
  ssl-default-bind-ciphers $supported_ciphers
  ssl-default-bind-options ssl-min-ver TLSv1.x ssl-max-ver TLSv1.x
  server-state-file /path/to/state_file
  tune.bufsize 40960
frontend frontend-80
	bind x.x.x.x:80 mss 1440 alpn h2,http/1.1
	mode http
	option httplog
	default_backend backend-sg
  acl acl-http-status-ok status 100:399
  filter opentracing id ot-test-ctx config /etc/haproxy/ctx/ot.cfg
  http-response ot-group ot-test-ctx http_response_group if acl-http-status-ok
  http-after-response ot-group ot-test-ctx http_after_response_group if !acl-http-status-ok
backend backend-sg
	mode  http
	option httpchk
	http-check send meth GET uri / ver HTTP/1.1 hdr Host something.com
	http-check expect status 200
	server my-server01 x.x.x.x:80 check port 80 maxconn 5000 enabled maxqueue 1
	server my-server02 x.x.x.x:80 check port 80 maxconn 5000 enabled maxqueue 1
	server my-server03 x.x.x.x:80 check port 80 maxconn 5000 enabled maxqueue 1
	server my-server04 x.x.x.x:80 check port 80 maxconn 5000 enabled maxqueue 1

Expected behavior

haproxy opentracing should run with chroot.

Steps to reproduce the behavior

Using the exact config I posted above should be sufficient to reproduce this issue.

Do you have any idea what may have caused this?

No

Do you have an idea how to solve the issue?

No

What is your configuration?

Posted above

Output of haproxy -vv and uname -a

HAProxy version 2.4.0-6cbbecf 2021/05/14 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2026.
Known bugs: http://www.haproxy.org/bugs/bugs-2.4.0.html
Running on: Linux 4.15.0-42-generic #45-Ubuntu SMP Thu Nov 15 19:32:57 UTC 2018 x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -Wall -Wextra -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  OPTIONS = USE_PCRE=1 USE_LINUX_TPROXY=1 USE_LINUX_SPLICE=1 USE_LIBCRYPT=1 USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1 USE_OT=1 USE_PROMEX=1
  DEBUG   =
Feature list : +EPOLL -KQUEUE +NETFILTER +PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H +GETADDRINFO +OPENSSL -LUA +FUTEX +ACCEPT4 -CLOSEFROM +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS +OT -QUIC +PROMEX -MEMORY_PROFILING
Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=16).
Built with OpenSSL version : OpenSSL 1.1.1h  22 Sep 2020
Running on OpenSSL version : OpenSSL 1.1.1h  22 Sep 2020
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with OpenTracing support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 7.3.0
Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using ‘proto’ keyword)
              h2 : mode=HTTP       side=FE|BE     mux=H2       flags=HTX|CLEAN_ABRT|HOL_RISK|NO_UPG
            fcgi : mode=HTTP       side=BE        mux=FCGI     flags=HTX|HOL_RISK|NO_UPG
       <default> : mode=HTTP       side=FE|BE     mux=H1       flags=HTX
              h1 : mode=HTTP       side=FE|BE     mux=H1       flags=HTX|NO_UPG
       <default> : mode=TCP        side=FE|BE     mux=PASS     flags=
            none : mode=TCP        side=FE|BE     mux=PASS     flags=NO_UPG
Available services : prometheus-exporter
Available filters :
	[  OT] opentracing
	[SPOE] spoe
	[CACHE] cache
	[FCGI] fcgi-app
	[COMP] compression
	[TRACE] trace

Linux hostname 4.15.0-42-generic #45-Ubuntu SMP Thu Nov 15 19:32:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[zaga00]

Hello @Kaminari-Y,

you did not put the configuration file for the opentracing filter (ot.cfg) but only for HAProxy. I will assume, the problem is that the opentracing library loaded from that configuration cannot load all the libraries it depends on after chroot. Check with the command 'ldd' the dependence of the opentracing library (it is the one specified in the ot.cfg configuration) and the availability of the required libraries after chroot.

Best regards.

[Kaminari-Y]

@zaga00

Thank you very much for replying. First let me share my opentracing filter, I only changed below 2 lines from the origal ot.cfg example file from addons/ot/test/ctx/ot.cfg

        config /etc/haproxy/ctx/cfg-jaeger.yml
        plugin /var/lib/haproxy/libjaegertracing_plugin.so

and cfg-jaeger.yml is also same as example addons/ot/test/ctx/cfg-jaeger.yml

My chroot directory is /var/lib/haproxy, I used ldd as you suggested, but the error is same

root@ray-haproxy:/var/lib/haproxy# ldd libjaegertracing_plugin.so
	linux-vdso.so.1 (0x00007ffc1b7ef000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f4315e87000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f4315ae9000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f43156f8000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f43165aa000)

root@ray-haproxy:/var/lib/haproxy# tree
.
├── bin
│   ├── bash
│   └── ls
├── busy-servers.sock
├── dev
│   └── log
├── haproxy.sock
├── lib
│   └── x86_64-linux-gnu
│       ├── libc.so.6
│       ├── libdl.so.2
│       ├── libm.so.6
│       ├── libpcre.so.3
│       ├── libpthread.so.0
│       ├── libselinux.so.1
│       └── libtinfo.so.5
├── lib64
│   └── ld-linux-x86-64.so.2
├── stats
├── usr
│   └── bin
│       └── ldd
├── v2-busy-servers.sock
└── var
    └── lib
        └── haproxy
            └── libpthread.so.0

I also did a ldd after chroot

root@ray-haproxy:/var/lib/haproxy# chroot . /bin/bash
bash-4.4# ldd libjaegertracing_plugin.so
	linux-vdso.so.1 (0x00007ffdcb184000)
	libpthread.so.0 => /var/lib/haproxy/libpthread.so.0 (0x00007f34e21c4000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f34e1e26000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f34e1a35000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f34e28e7000)
bash-4.4#

Maybe I didn't do enough what you suggested, sorry about that and would you guide me a little more.

Thank you very much!

[zaga00]

Hello @Kaminari-Y,

the problem is that HAProxy also executes chdir("/") during chroot process, so the paths written in the opentracing configuration are no longer correct. This could be easily solved by writing the absolute path when using the 'config' and 'plugin' keywords, but the problem remains that the validity of these paths is also checked before the chroot process. This means that one will need to commit one small patch to correct this. :)

It will probably be resolved tomorrow. Thank you for reporting the bug.

Best regards.

[Kaminari-Y]

Hello @zaga00

Understood the issue now, thank you very much for help!
Best Regards.

[wtarreau]

Miroslav, if we're facing such an issue, there's a deeper problem. It means that some files are accessed at run time, which must never happen. Probably that a few things are missing in the filter's initialization code.

Some libraries are a bit tricky because they're using lazy loading. For example, openssl only opens /dev/urandom on first crypto use, which is why we're performing an early dummy call to force it to initialize. You may need to perform a few early calls to the libs during init to make sure all needed files are completely loaded before chrooting.

[zaga00]

Hello Willy,

the problem is the initialization of the opentracing API library, which starts an additional thread through which the opentracing communicates with the tracer (the server on which the tracing data is stored). Initializing the API library requires as arguments the path to the plugin library and its configuration.

Initialization was done before the chroot call but there was a problem if HAProxy is used in daemon mode. Then something in the process of daemonization killed the opentracing thread and the filter could no longer work. That was the reason i put the initialization of the opentracing library after switching the HAProxy to daemon mode (and this is also after the chroot process).

Perhaps we should further investigate what it is that kills the opentracing thread when switching HAProxy to daemon mode.

[zaga00]

And in fact, the only access to these files is achieved only once at the beginning of the HAProxy process, in the initialization of threads. After this initialization, no access to the file system is performed.

[zaga00]

Hello @Kaminari-Y,

the issue you reported has been resolved, please test if it is working properly now. You don't need to specify the full path to the tracer configuration and plugin library, in fact the only change is that you need to add the keyword 'chroot' to the haproxy configuration.

Best regards.

[VigneshSP94]

@wtarreau @zaga00 is this fix merged to any release or should we take the committed branch and test it?

[wtarreau]

It's in the latest 2.4 branch, but not yet in any released version. We should emit 2.4.1 this week so it will be there :-)

[Kaminari-Y]

@wtarreau @zaga00 @VigneshSP94

I've tested 2.4.1 and the issue is gone.
Works fine for me !!!

Thank you guys very much for helping !!

[wtarreau]

Thank you for the test! I'm closing the issue then.