Rate limiting using map seems to be misbehaving

[madhanrajrm]

Detailed Description of the Problem

We are using haproxy v 2.2

i have this below frontend which handles few incoming API's so i would like to control the few particular API in my frontend so that i can control my outgoing requests to outbound.

frontend http

bind 127.0.0.1:84 tfo accept-proxy
acl is_ssl fc_rcvd_proxy
option nolinger
stick-table type binary len 8 size 100k expire 10s store http_req_rate(1s)
http-request track-sc0 path
acl remoteDestinations_ep1 path_end /remoteDestinations
http-request set-var(req.rate_limit) path,map_beg(/usr/haproxyrates.map,500) unless remoteDestinations_ep1
http-request set-var(req.rate_limit) int(10) if remoteDestinations_ep1
http-request set-var(req.request_rate) path,table_http_req_rate()
acl rate_abuse var(req.rate_limit),sub(req.request_rate) lt 0
#http-request deny deny_status 429 if rate_abuse
http-request deny deny_status 503 if rate_abuse remoteDestinations_ep1
use_backend slowservers if rate_abuse
default_backend usemultipleports.

below is my contents of haproxyrates.map file
/ann/private/users 1
/ann/version 100
/ann/servers 100
/ann/private/options/userPolicy 80
/ann/private/user 80
/ann/user 30
/ann/clusterUser 10

So with my current understanding of the config what it should do was

1. When /ann/user/23/remoteDestinations API hits haproxy it just always returns with 503 when i trigger this API along with other api’s like /ann/clusterUser , /ann/user, /ann/private/user ,/ann/servers rate limits.

2. For eg., when at any given second if i send 4 /ann/private/users API ( which has rate limited to 1 ) and 4 /ann/user/remoteDestinations i see all 4 remoteDestination API got returned with 503 responses but ideally it should have served.

Since rate_abuse variable should be set based on looking the path rather than all the entries in the map.

Expected Behavior

Explained in the description. I dont feel rate limit map works how it should be when we need to handle multiple API's

Steps to Reproduce the Behavior

shared the steps in description

Do you have any idea what may have caused this?

No response

Do you have an idea how to solve the issue?

No response

What is your configuration?

frontend http

bind 127.0.0.1:84 tfo accept-proxy
  acl is_ssl fc_rcvd_proxy
  option nolinger
  stick-table  type binary  len 8  size 100k  expire 10s  store http_req_rate(1s)
  http-request track-sc0 path
  acl remoteDestinations_ep1 path_end /remoteDestinations
  http-request set-var(req.rate_limit)  path,map_beg(/usr/haproxyrates.map,500) unless remoteDestinations_ep1
  http-request set-var(req.rate_limit) int(10) if remoteDestinations_ep1
  http-request set-var(req.request_rate)  path,table_http_req_rate()
  acl rate_abuse var(req.rate_limit),sub(req.request_rate) lt 0
  #http-request deny deny_status 429 if rate_abuse
  http-request deny deny_status 503 if rate_abuse remoteDestinations_ep1
  use_backend slowservers if rate_abuse
  default_backend usemultipleports.

below is my contents of haproxyrates.map file
/ann/private/users 1
/ann/version 100
/ann/servers 100
/ann/private/options/userPolicy 80
/ann/private/user 80
/ann/user 30
/ann/clusterUser 10

Output of haproxy -vv

HA-Proxy version 2.2.32-4081d5a 2023/12/19 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2025.
Known bugs: http://www.haproxy.org/bugs/bugs-2.2.32.html
Running on: Linux 3.10.0-1160.90.1.el7.x86_64 #1 SMP Mon Jun 12 15:16:58 IST 2023 x86_64
Build options :
  TARGET  = custom
  CPU     = generic
  CC      = gcc
  CFLAGS  = -m32
  OPTIONS = USE_EPOLL=1 USE_THREAD=1 USE_LINUX_SPLICE=1 USE_OPENSSL=1 USE_CPU_AFFINITY=1
  DEBUG   =

Feature list : -51DEGREES -ACCEPT4 -BACKTRACE -CLOSEFROM +CPU_AFFINITY -CRYPT_H -DEVICEATLAS -DL +EPOLL -EVPORTS -FUTEX -GETADDRINFO -KQUEUE -LIBCRYPT +LINUX_SPLICE -LINUX_TPROXY -LUA -NETFILTER -NS -OBSOLETE_LINKER +OPENSSL -PCRE -PCRE2 -PCRE2_JIT -PCRE_JIT +POLL -PRCTL -PRIVATE_CACHE -PTHREAD_PSHARED -RT -SLZ -STATIC_PCRE -STATIC_PCRE2 -SYSTEMD -TFO +THREAD -THREAD_DUMP -TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=32, default=2).
Built with OpenSSL version : CiscoSSL 1.1.1v.7.2.539
Running on OpenSSL version : CiscoSSL 1.1.1v.7.2.539
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built without compression support (neither USE_ZLIB nor USE_SLZ are set).
Compression algorithms supported : identity("identity")
Built with transparent proxy support using: IP_TRANSPARENT IP_FREEBIND
Built without PCRE or PCRE2 support (using libc's regex instead)
Encrypted password support via crypt(3): no
Built with gcc compiler version 4.8.5 20150623 (Red Hat 4.8.5-39)

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
            fcgi : mode=HTTP       side=BE        mux=FCGI
       <default> : mode=HTTP       side=FE|BE     mux=H1
              h2 : mode=HTTP       side=FE|BE     mux=H2
       <default> : mode=TCP        side=FE|BE     mux=PASS

Available services : none

Available filters :
        [SPOE] spoe
        [COMP] compression
        [TRACE] trace
        [CACHE] cache
        [FCGI] fcgi-app

Last Outputs and Backtraces

No response

Additional Information

No response

[Darlelet]

From your config:

stick-table type binary len 8 size 100k expire 10s store http_req_rate(1s)

Since table_http_req_rate() uses path as unique key, and that your setup involves long paths (more than 8 chars long), my guess is that 8 is too small to contain your various paths.. thus all paths beginning with /ann/use* will match with the same stick table entry (and increase the same http request counter)

Please try increasing the stick table's key length to a relevant value for your use-case so that all your paths end up being stored as distinct keys, or simply hash paths using a hashing function to store the key as "unique" fixed-length integers, like this:

frontend http

bind 127.0.0.1:84 tfo accept-proxy
acl is_ssl fc_rcvd_proxy
option nolinger
http-request set-var(req.path_key) path,xxh32 # convert path to integer using hashing function

stick-table type integer size 100k expire 10s store http_req_rate(1s) # key has integer type (lower memory footprint and faster lookup)
http-request track-sc0 var(req.path_key) # track using hash
acl remoteDestinations_ep1 path_end /remoteDestinations
http-request set-var(req.rate_limit) path,map_beg(/usr/haproxyrates.map,500) unless remoteDestinations_ep1
http-request set-var(req.rate_limit) int(10) if remoteDestinations_ep1
http-request set-var(req.request_rate) var(req.path_key),table_http_req_rate() # lookup using path
acl rate_abuse var(req.rate_limit),sub(req.request_rate) lt 0
#http-request deny deny_status 429 if rate_abuse
http-request deny deny_status 503 if rate_abuse remoteDestinations_ep1
use_backend slowservers if rate_abuse
default_backend usemultipleports.

Also please consider upgrading to the next LTS version because 2.2 will soon reach EOL.

[madhanrajrm]

Thanks @Darlelet that made more sense , we could see it is working fine now with the full path as unique key . we got carried away by the documentation of stick table

stick-table type {ip | integer | string [len ] | binary [len ]} size [expire ] [nopurge] [peers ] [store <data_type>]*

we thought len were corresponding to binary type and not for str.

Yes we are waiting for official release of 3.0 to pick .