Order of configuration changes after Dataplane API commit

[cahitonur]

Hi,
I’m using Dataplane API to update HAProxy configuration and everything works well except the new version of configuration has a different order of directives under backends and frontends. To give an example, let’s say I have this pseudo backend block;

backend be_static
  mode http
  http-request set-header Host example.net
  http-response add-header Set-Cookie my_cookie=%[var(txn.my_cookie)]
  server example example.net ssl verify none sni str(example.net)

After adding another backend with Dataplane API my be_static backend ends up in this order;

backend be_static
  mode http
  http-request set-header Host example.net
  server example example.net ssl verify none sni str(example.net)
  http-response add-header Set-Cookie my_cookie=%[var(txn.my_cookie)]

and it breaks the functionality. Not only backends but my frontend configurations suffer from this change of order too after an API commit. For example;

capture request header Referer len 200
capture request header User-Agent len 200
log-format %ci\ -\ %{+Q}r\ %ST\ %U\ %{+Q}[capture.req.hdr(0)]\ %{+Q}[capture.req.hdr(1)]\ %{+Q}b\ %{+Q}s
use_backend %[base,lower,map_beg(api.map)]-http if { path_beg /api }
use_backend %[base,lower,map_beg(http.map)]

becomes

log-format %ci\ -\ %{+Q}r\ %ST\ %U\ %{+Q}[capture.req.hdr(0)]\ %{+Q}[capture.req.hdr(1)]\ %{+Q}b\ %{+Q}s
use_backend %[base,lower,map_beg(api.map)]-http if { path_beg /api }
use_backend %[base,lower,map_beg(http.map)]
capture request header Referer len 200
capture request header User-Agent len 200

Any idea why this happens or a workaround to prevent it happening?

# haproxy -vv
HA-Proxy version 2.0.14-1ppa1~bionic 2020/04/16 - https://haproxy.org/
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -O2 -fdebug-prefix-map=/build/haproxy-vOKb8p/haproxy-2.0.14=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wno-stringop-overflow -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_SYSTEMD=1

Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=2).
Built with OpenSSL version : OpenSSL 1.1.1  11 Sep 2018
Running on OpenSSL version : OpenSSL 1.1.1  11 Sep 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE2 version : 10.31 2018-02-12
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with the Prometheus exporter as a service

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTX        side=FE|BE     mux=H2
              h2 : mode=HTTP       side=FE        mux=H2
       <default> : mode=HTX        side=FE|BE     mux=H1
       <default> : mode=TCP|HTTP   side=FE|BE     mux=PASS

Available services :
	prometheus-exporter

Available filters :
	[SPOE] spoe
	[COMP] compression
	[CACHE] cache
	[TRACE] trace

[mjuraga]

This happens because of the way how our config-parser works. It orders them in a specific order where it bunches up same configuration directives in order to preserve their respective ordering.

I see how this may be an issue in your case of frontend, because we do not parse capture directive and it ends at the bottom of the section, so you can use http-request capture instead. Which would still leave it below the log-format directive so this is a bug.

Also http-response directives should come in directly after http-request directives, so this is also a bug. We will fix this.

[mjuraga]

Looking again and testing your examples I don't see a functionality issue with them.

[cahitonur]

The problem with first example is that I am not able to set a cookie if order changes. HAProxy ignores the line where I set cookie and the cookie is used for a conditional routing by checking its value. Hence my conditional routing is not working.
Second example is not a full one but with order changed, log format is getting ignored. Indeed it's not a functionality problem and I gave that example just to show it's happening on both backend and frontend.

[Mo3m3n]

Hi @cahitonur
The order of http-response add-header Set-Cookie my_cookie=%[var(txn.my_cookie)] does not matter in your case (before or after the server line). It should work in both cases.
If it is not working that's probably because the variable txn.my_cookie is not set and in such case it won't work on any order.

[cahitonur]

Hi @Mo3m3n, thanks for the input. I tested everything once again. There are no functionality problems indeed. I think I was not getting expected routing because of cache.
Log format issue is still going on though. With order changed HAProxy ignores our custom format given in config and uses default format.

[Mo3m3n]

I have tested this exact snippet

log-format %ci\ -\ %{+Q}r\ %ST\ %U\ %{+Q}[capture.req.hdr(0)]\ %{+Q}[capture.req.hdr(1)]\ %{+Q}b\ %{+Q}s
use_backend %[base,lower,map_beg(api.map)]-http if { path_beg /api }
use_backend %[base,lower,map_beg(http.map)]
capture request header Referer len 200
capture request header User-Agent len 200

And it works as expected meaning that I have the expected log format, are you sure the problem is not elsewhere

[cahitonur]

@Mo3m3n Hi, sorry for the late response. Let me share the exact frontend config.

I start with this:

frontend http-requests
  mode http
  bind :80
  bind :443 ssl crt /etc/haproxy/certs/ssl.pem alpn h2,http/1.1
  option forwardfor
  log /dev/log local0
  option httplog
  capture request header Referer len 200
  capture request header User-Agent len 200
  log-format %ci\ -\ %{+Q}r\ %ST\ %U\ %{+Q}[capture.req.hdr(0)]\ %{+Q}[capture.req.hdr(1)]\ %{+Q}b\ %{+Q}s
  http-request redirect scheme https code 301 if !{ ssl_fc }
  http-response set-header Strict-Transport-Security max-age=63072000
  use_backend %[base,lower,map_beg(/etc/haproxy/maps/api.map)]-http if { path_beg /api }
  use_backend %[base,lower,map_beg(/etc/haproxy/maps/http.map)]

and end up with the snippet below after an update over API:

frontend http-requests
  mode http
  bind :80
  bind :443 ssl crt /etc/haproxy/certs/ssl.pem alpn h2,http/1.1
  log-format %ci\ -\ %{+Q}r\ %ST\ %U\ %{+Q}[capture.req.hdr(0)]\ %{+Q}[capture.req.hdr(1)]\ %{+Q}b\ %{+Q}s
  log /dev/log local0
  option forwardfor
  option httplog
  http-request redirect scheme https code 301 if !{ ssl_fc }
  use_backend %[base,lower,map_beg(/etc/haproxy/maps/api.map)]-http if { path_beg /api }
  use_backend %[base,lower,map_beg(/etc/haproxy/maps/http.map)]
  http-response set-header Strict-Transport-Security max-age=63072000
  capture request header Referer len 200
  capture request header User-Agent len 200

Hence log format is getting ignored.

There a few other lines responsible of some cookie setup but I believe they are unrelated. I didn't try all the possibilities but I suspect the order of log /dev/log local0 or option httplog is causing this problem. I don't want to move them to defaults as I'm interested in logging only this frontend and not sure if there is another way to configure selective logging which will solve this problem but with this configuration my custom log format is ignored and I'm seeing default format in logs.

[mjuraga]

Hey @cahitonur , you are using both option httplog and log-format and the latest to appear in the configuration will override the other. So here if you want to use your custom log format it is ok for you to just omit the option httplog from your configuration. From the HAProxy documentation the option httplog is the same as:

At the moment, the default HTTP format is defined this way :

    log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC \
                %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"

[cahitonur]

@mjuraga Removing option httplog solved the problem. Thank you!