question

[Arnold1]

hi,
i have the following architecture with have http trafffic (POST request/response):

ip master_server:  74.120.111.152
ip process_server: 10.211.148.15, service: calcData runs on port 8100
ip qa_server: 10.211.148.16, service: calcData runs on port 8100

master_server ---- process_server 
                  (service: calcData)

here is how i would like to use haproxy. would run on process_server read the incomming traffic from master_server and forward the traffic to:

• service: calcData running on process_server
• service: calcData running on qa_server

master_server ----- process_server ----- qa_server
                       haproxy <----->  service: calcData
                          ^
                          |
                          v
                    service: calcData

as far i understand you need spoa-mirror to mirror it since haproxy is not able to do so.
here is how i configured spoa-mirror.

./src/spoa-mirror -r0 -u"http://10.211.148.16:8100/"

i use HA-Proxy version 2.0.0-3f0b1d-9 2019/06/19 on debian (make TARGET=custom CPU=native USE_PCRE=1 USE_LINUX_SPLICE=1 USE_LINUX_TPROXY=1):

here is how my haproxy config looks like (master_server needs to send to 10.211.148.15:8000):
how can configure the haproxy for low latency (1-20ms response time) and 60k QPS? Please give me some feedback. about those settings i currently use or which ones i could add. should i enable http-keep-alive?

global
    maxconn 1000
    #daemon
    #stats timeout 30s

defaults
    # Enable HTTP keep-alive from client to server.
    #option http-keep-alive

    # Maximum time to wait for a connection attempt to a server to succeed.
    timeout connect 1s

    # Inactivity timeout on the client side for half-closed connections.
    # Applies when the client is expected to acknowledge or send data 
    # while one direction is already shut down.
    #timeout client-fin 5s

    # Maximum inactivity time on the client side.
    # Applies when the client is expected to acknowledge or send data.
    timeout client 1s
    
    # Inactivity timeout on the client side for half-closed connections.
    # Applies when the client is expected to acknowledge or send data 
    # while one direction is already shut down.
    #timeout server-fin 5s
    
    # Maximum inactivity time on the server side.
    timeout server 5s

    # Clients should send their full http request in 100ms.
    #timeout http-request 100ms

    # Maximum allowed time to wait for a new HTTP request to appear.
    #timeout http-keep-alive 300s

    stats enable
    stats realm Haproxy\ Statistics  # Title text for popup window
    stats uri /haproxy_stats
    stats realm Haproxy\ Statistics
    stats refresh 5s

frontend http-in
    # Frontend bound on all network interfaces on port 8000
    bind *:8000
    # default traffic mode is http
    # mode is overwritten in case of tcp services
    mode http
    # default default_backend. This allows custom default_backend in frontends
    default_backend calc-service

backend calc-service
    #option httpclose
    #option forwardfor
    mode http
    server localhost:8100 localhost:8100

i have a few questions:

• how much traffic can haproxy handle? is it able to handle 60k QPS?
• i have traffic with low latency:
  the latency between master_server and process server is 1-20ms
  haproxy would run on process_server and forward the traffic to qa_server
  is haproxy able to handle such low latency?
• whats the overhead to include haproxy for forwarding http traffic to qa_server?

[chadlavoie]

It seems there are two independent questions here which need to be handled separately. First is getting HAProxy cranking at your requirements, and the second is getting HAProxy with the mirror SPOA cranking without harming the performance.
Given that 60k QPS is doable but on the high side of normal I'd highly recommend making sure HAProxy is working at your desired request rate and latency before adding SPOA to the mix. While adding SPOA shouldn't add delays given the volume of traffic I'd not count on that without having a fall-back and the ability to debug both of these things separately.
Some general things that you will want to think about for this:

• Having a single point of failure and no ability to scale at this connection rate isn't advisable, I'd recommend using BGP to split traffic across multiple HAProxy servers
• Source ports are going to be a problem at 60k QPS depending on connection reuse and connection lifetime. Without knowing what type of traffic you are dealing with I can't suggest too much, though looking at connection pooling from HAProxy to the backends is a good idea.
• Ensure that sysctl is properly tuned. Exactly how depends on your use case, but net.ipv4.tcp_tw_reuse, net.ipv4.tcp_rmem, net.ipv4.tcp_wmem, net.ipv4.ip_local_port_range are ones that need to be tuned in almost every case (and the IPv6 equivalents of applicable)
• The timing fields in the HAProxy logs give times in milliseconds for each phase of the request to see why a given request is taking as long as it is. Also see profiling.tasks to see how long internal HAProxy processing is taking for your configuration.
• When you get to SPOA you can set up multiple SPOA agents so if it gets loaded at your desired sampling rate you can set up multiple of them.