Custom LB Backend Corrupts HAProxy Config on Service Stop

[jmloeffler]

Using Cattle, if you create a load balancer with custom backend names in order to customize the configuration, the haproxy config is corrupted and won't reload if one of the backends has no servers available. To replicate:

1. create a stack with two services service1 and service2
2. create a load balancer exposing ports 81 and 82 and custom backend names backend1 and backend2
3. customize the haproxy config with something simple as a test:

backend backend1
server $IP check

backend backend2
server $IP check

4. View the load balancer logs and it will say:

4/5/2017 3:59:59 PMtime="2017-04-05T20:59:59Z" level=info msg="Starting Rancher LB service"
4/5/2017 3:59:59 PMtime="2017-04-05T20:59:59Z" level=info msg="LB controller: rancher"
4/5/2017 3:59:59 PMtime="2017-04-05T20:59:59Z" level=info msg="LB provider: haproxy"
4/5/2017 3:59:59 PMtime="2017-04-05T20:59:59Z" level=info msg="starting rancher controller"
4/5/2017 3:59:59 PMtime="2017-04-05T20:59:59Z" level=info msg="Healthcheck handler is listening on :10241"
4/5/2017 4:00:00 PMtime="2017-04-05T21:00:00Z" level=info msg=" -- starting haproxy\n * Starting haproxy haproxy\n ...done.\n"
4/5/2017 4:00:01 PMtime="2017-04-05T21:00:01Z" level=info msg=" -- reloading haproxy config with the new config changes\n * Reloading haproxy haproxy\n[WARNING] 094/210001 (56) : config : 'option forwardfor' ignored for proxy 'default' as it requires HTTP mode.\n[WARNING] 094/210001 (58) : config : 'option forwardfor' ignored for proxy 'default' as it requires HTTP mode.\n ...done.\n"

5. Now stop service1 and view the load balancer logs again and errors will appear:

4/5/2017 4:01:12 PMtime="2017-04-05T21:01:12Z" level=info msg=" -- reloading haproxy config with the new config changes\n * Reloading haproxy haproxy\n[ALERT] 094/210112 (82) : parsing [/etc/haproxy/haproxy.cfg:31] : character '$' is not permitted in 'backend' name 'backend1_$IP'.\n[ALERT] 094/210112 (82) : parsing [/etc/haproxy/haproxy.cfg:32] : unknown keyword 'check' in 'backend' section\n[ALERT] 094/210112 (82) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg\n[WARNING] 094/210112 (82) : config : 'option forwardfor' ignored for backend 'backend1_$IP' as it requires HTTP mode.\n[WARNING] 094/210112 (82) : config : 'option forwardfor' ignored for proxy 'default' as it requires HTTP mode.\n[ALERT] 094/210112 (82) : Fatal errors found in configuration.\n ...fail!\n"
4/5/2017 4:01:12 PMtime="2017-04-05T21:01:12Z" level=error msg="Failed to apply lb config on provider: error reloading -- reloading haproxy config with the new config changes\n * Reloading haproxy haproxy\n[ALERT] 094/210112 (82) : parsing [/etc/haproxy/haproxy.cfg:31] : character '$' is not permitted in 'backend' name 'backend1_$IP'.\n[ALERT] 094/210112 (82) : parsing [/etc/haproxy/haproxy.cfg:32] : unknown keyword 'check' in 'backend' section\n[ALERT] 094/210112 (82) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg\n[WARNING] 094/210112 (82) : config : 'option forwardfor' ignored for backend 'backend1_$IP' as it requires HTTP mode.\n[WARNING] 094/210112 (82) : config : 'option forwardfor' ignored for proxy 'default' as it requires HTTP mode.\n[ALERT] 094/210112 (82) : Fatal errors found in configuration.\n ...fail!\n: exit status 1"

From this point forward, haproxy in the load balancers will not accept any config updates until all of the customized backend services are up and running again. In the haproxy config in the LB container(s) you will find the invalid config section:

backend backend1_$IP
check

It seems like if there are no servers available the parsing logic doesn't know what to do with it and just stuffs it into the config like that. This prevents HAProxy from reloading any new config changes, including new services that spin up or removing servers that have gone down. Effectively, haproxy is stuck with the last good config it had before the customized-backend service went down. You can add new services, delete them, stop them, et cetera and the haproxy config in effect will remain unchanged. I've tested this back to Rancher 1.2 (when the custom backends were introduced) and it seems to be the same behavior there.

---

Useful	Info
Versions	Rancher v1.5.3 Cattle: v0.177.10 UI: v1.5.8
Access	Disabled
Orchestration	Cattle
Route	service.containers

[prachidamle]

Thank you for such detail steps for the bug @jmloeffler!

[sangeethah]

Tested with v1.6.0-rc3 and rancher/lb-service-haproxy:v0.7.1

Created LB service with custom config for named backends using the following yml.

Stopped the services that are targets to the named backend.

HA proxy reload succeed. No error reported on reload.

version: '2'
services:
  test322355:
    image: sangeetha/testnewhostrouting:latest
  LB-test678536:
    image: rancher/lb-service-haproxy:v0.7.1
    ports:
    - 1923:1923/tcp
    labels:
      io.rancher.container.agent.role: environmentAdmin
      io.rancher.container.create_agent: 'true'
  test146807:
    image: sangeetha/testnewhostrouting:latest

version: '2'
services:
  test322355:
    scale: 2
    start_on_create: true
  LB-test678536:
    scale: 1
    start_on_create: true
    lb_config:
      certs: []
      config: |-
        backend myrule1
        server $$IP check
      port_rules:
      - backend_name: myrule1
        hostname: www.abc1.com
        path: /service1.html
        protocol: http
        service: test146807
        source_port: 1923
        target_port: 80
      - hostname: www.abc2.com
        path: /service2.html
        protocol: http
        service: test322355
        source_port: 1923
        target_port: 80
    health_check:
      healthy_threshold: 2
      response_timeout: 2000
      port: 42
      unhealthy_threshold: 3
      initializing_timeout: 60000
      interval: 2000
      reinitializing_timeout: 60000
  test146807:
    scale: 2
    start_on_create: true