server-state-file issue with "show servers state" command

[agirot]

Detailed description of the problem

Since few release (1.7?), the "show servers state" command returns the port of each backend server and the "load-server-state-from-file" parameter overwrites all ports of the config file.

Expected behavior

Mismatch between HAProxy's documentation and HAProxy's behavior:

https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#3.1-server-state-file
"Specifies the path to the file containing state of servers. If the path starts
with a slash ('/'), it is considered absolute, otherwise it is considered
relative to the directory specified using "server-state-base" (if set) or to
the current directory. Before reloading HAProxy, it is possible to save the
servers' current state using the stats command "show servers state". The
output of this command must be written in the file pointed by . When
starting up, before handling traffic, HAProxy will read, load and apply state
for each server found in the file and available in its current running
configuration. See also "server-state-base" and "show servers state",
"load-server-state-from-file" and "server-state-file-name"

Steps to reproduce the behavior

1. Add server-state-file /root/status and load-server-state-from-file global in config file
2. Run socat /run/haproxy/admin.sock - <<< "show servers state" > /root/status
3. systemctl reload haproxy
4. Network traffic still send on previous server port

Do you have any idea what may have caused this?

show servers state return port of each servers.
I think the server-state-file param overwrite the port defined in the configuration with the port of servers state file

Do you have an idea how to solve the issue?

Don't use the port number written in the server status file ?

What is your configuration?

global
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
daemon
server-state-file /root/status
debug

defaults
        log     global
        timeout connect 5000ms
        timeout client 5m
        timeout server 5m
        timeout tunnel 15m
        option splice-auto
        load-server-state-from-file global

frontend test
  bind 0.0.0.0:443
  mode tcp
  option tcplog
  maxconn 20000
  default_backend test_backend

backend test_backend
  mode tcp
  balance roundrobin
  option tcp-check
  timeout check 10000ms
  timeout server 21600000ms
  timeout tunnel 21600000ms
  timeout connect 5000ms

  server srv_0 X.X.X.X:443 check port 443 inter 1000ms fall 2

Output of haproxy -vv and uname -a

HA-Proxy version 2.1.10-1ppa1~bionic 2020/11/08 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2021.
Known bugs: http://www.haproxy.org/bugs/bugs-2.1.10.html
Running on: Linux 4.15.0-109-generic #110-Ubuntu SMP Tue Jun 23 02:39:32 UTC 2020 x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -O2 -fdebug-prefix-map=/build/haproxy-q3dkrS/haproxy-2.1.10=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wno-stringop-overflow -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_SYSTEMD=1

Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +BACKTRACE +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -CLOSEFROM -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=2).
Built with OpenSSL version : OpenSSL 1.1.1  11 Sep 2018
Running on OpenSSL version : OpenSSL 1.1.1  11 Sep 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.31 2018-02-12
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with the Prometheus exporter as a service

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTTP       side=FE|BE     mux=H2
            fcgi : mode=HTTP       side=BE        mux=FCGI
       <default> : mode=HTTP       side=FE|BE     mux=H1
       <default> : mode=TCP        side=FE|BE     mux=PASS

Available services :
	prometheus-exporter

Available filters :
	[SPOE] spoe
	[CACHE] cache
	[FCGI] fcgi-app
	[TRACE] trace
	[COMP] compression

Linux 6b202020-0856-4677-886d-b5512e269db8 4.15.0-109-generic #110-Ubuntu SMP Tue Jun 23 02:39:32 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[wtarreau]

Sorry for missing this one, and 2.1 is now dead, but here's a short response before closing.

server-state-file was designed to allow the passing of previous dynamically changed settings from an older process to a new one. The problem in its design is that it didn't evolve sufficiently to make the distinction between deliberate changes and previous config values, so some settings have a higher priority form the state file because they're known to be changeable by the CLI while others have a higher priority in the config file.

It was already identified that this file ought to be changed so that all values have both the initial and current values, and that newly loaded config values always override the state file if they differ from the previous initial ones. It's not always as trivial as it seems however (think about address resolution which may result in different addresses, and where fqdn might also have to be returned). But this is definitely something that has been identified as having to be reworked for the few previous versions already but was left behind more urgent things :-/