Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow custom options #403

Merged
merged 5 commits into from Aug 18, 2017

Conversation

@sadlil
Copy link
Contributor

commented Aug 18, 2017

Fix #343

Supports all valid options for defaults section of HAProxy config from the list from here
https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#4.2-option%20abortonclose

Expects a json encoded map
ie:

ingress.appscode.com/default-option": '{"http-keep-alive": "true", "dontlognull": "true", "clitcpka": "false"}'

This will be appended in the defaults section of HAProxy as

   option http-keep-alive
   option dontlognull
   no option clitcpka

It is upto user to provide valid configurations. Providing invalid options will not run HAProxy. The log will display something like

daemon.err: Aug 18 06:14:41 reloader:  [ALERT] 229/061441 (37) : parsing [/etc/haproxy/haproxy.cfg:21]: negation/default is not supported for option 'invalid-options-false'.
daemon.err: Aug 18 06:14:41 reloader: [ALERT] 229/061441 (37) : parsing [/etc/haproxy/haproxy.cfg:22] : unknown option 'invalid-options-true'.
daemon.err: Aug 18 06:14:41 reloader: [ALERT] 229/061441 (37) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
daemon.err: Aug 18 06:14:41 reloader: [ALERT] 229/061441 (37) : Fatal errors found in configuration.
daemon.err: Aug 18 06:14:41 reloader: I0818 06:14:41.308405      18 mount.go:140] failed to run cmd

@@ -270,6 +285,23 @@ func (r Ingress) Timeouts() map[string]string {
return ans
}

func (r Ingress) Options() map[string]bool {

This comment has been minimized.

Copy link
@tamalsaha

tamalsaha Aug 18, 2017

Member

HAProxyOptions()

# Disable logging of null connections (haproxy connections like checks).
# This avoids excessive logs from haproxy internals.
option dontlognull
# Defaults Options provided by user

This comment has been minimized.

Copy link
@tamalsaha

tamalsaha Aug 18, 2017

Member

# Defaults Options provided by user these are not always provided by the user. Remove this comment. Instead add a link to HAProxy docs.

@tamalsaha tamalsaha merged commit cc85f0e into master Aug 18, 2017
@tamalsaha tamalsaha deleted the haproxy-options branch Aug 18, 2017
@greg-jaunt

This comment has been minimized.

Copy link

commented Aug 30, 2017

@sadlil the example above and in the comment in the source code is incorrect. You must enclose the options in single quotes, like so:

ingress.appscode.com/default-option": '{"http-keep-alive": "true", "dontlognull": "true", "clitcpka": "false"}'

@sadlil sadlil referenced this pull request Aug 31, 2017
@greg-jaunt

This comment has been minimized.

Copy link

commented Aug 31, 2017

After more testing, I find this implementation very limiting. It only allows settings that start with the word "option" and it seems to be somewhat arbitrary which keywords HAProxy has chosen to use "option" with. This now allows me to specify "option http-keep-alive" but, for example, does not allow me to specify "balance leastconn".

@tamalsaha

This comment has been minimized.

Copy link
Member

commented Aug 31, 2017

We have been talking about this internally. There is also a similar request in #439 . The general issue is that HAProxy has too many options and converting them all into annotations seems the wrong way to do it. One way we talked were just let users specify the haproxy.conf. That will allow anyone to do any customization they see fit. The question obviously is what will that look like?

One idea I have is allow users to provide the HAProxy template via a ConfigMap. We currently have a built-in template https://github.com/appscode/voyager/blob/master/pkg/haproxy/template.go#L31 . The user provided template must have the parameters that voyager operator uses to inject actual pod ips etc. But the any option, etc can be left to users to modify to their heart's content.

What do folks guys think about that?

cc: @julianvmodesto

@greg-jaunt

This comment has been minimized.

Copy link

commented Aug 31, 2017

If you wanted to use ConfigMaps for the config, how would you deal with the backend section, which can be dynamic when voyager pods change nodes? Perhaps you would need a ConfigMap per section (defaults, backend, frontend).

FWIW, I was able to work around this for "balance leastconn" by adding it as a backendRule: in the ingress.

@greg-jaunt

This comment has been minimized.

Copy link

commented Aug 31, 2017

I suppose, alternatively, Voyager could start with the user-provided ConfigMap config and modify it either smartly somehow or based on templated variables that the user would have to add to their ConfigMap.

@tamalsaha

This comment has been minimized.

Copy link
Member

commented Aug 31, 2017

I'm afraid that this is one of those situations where there are no simple answer.

I think the way we do it we provide the templates as separate files. Each variable in https://github.com/appscode/voyager/blob/master/pkg/haproxy/template.go#L41 becomes a separate template file. The root template loads the individual pieces. All these options are the root template section, if you want to modify the global / defaults section.

So, users can replace the root template to inject their custom options and keep the other templates as-is. Voyager should be able to inject the dynamic parts easily with that.

@tamalsaha

This comment has been minimized.

Copy link
Member

commented Aug 31, 2017

This is the root template. Users replace this with their custom version.

# HAProxy configuration generated by https://github.com/appscode/voyager
# DO NOT EDIT!
global
	daemon
	stats socket /tmp/haproxy
	server-state-file global
	server-state-base /var/state/haproxy/
	maxconn 4000
	# log using a syslog socket
	log /dev/log local0 info
	log /dev/log local0 notice
	tune.ssl.default-dh-param 2048
	ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK
defaults
	log global
	# https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#4.2-option%20abortonclose
	# https://github.com/appscode/voyager/pull/403
	{{- range $k, $v := .OptionsDefaults }}
	{{ if not $v }}no {{ end }}option {{ $k -}}
	{{ end }}
	# Timeout values
	{{- range $k, $v := .TimeoutDefaults }}
	timeout {{ $k }} {{ $v -}}
	{{ end }}
	# default traffic mode is http
	# mode is overwritten in case of tcp services
	mode http
{{- range $resolver := .DNSResolvers }}
{{ template "dns-resolver" $resolver -}}
{{ end }}
{{- if .Stats }}
{{ template "stats" .Stats -}}
{{ end -}}
{{- range $svc := .HTTPService }}
{{- template "http-frontend" $svc  }}
{{ template "http-backend" $svc  }}
{{ end -}}
{{- range $svc := .TCPService }}
{{ template "tcp-frontend" $svc }}
{{ template "tcp-backend" $svc }}
{{ end -}}
{{- if and (not .HTTPService) .DefaultBackend }}
{{ template "default-frontend" .SharedInfo }}
{{ end }}
{{- if .DefaultBackend }}
{{ template "default-backend" .SharedInfo }}
{{ end }}
`))
@tamalsaha

This comment has been minimized.

Copy link
Member

commented Sep 2, 2017

I have got a pr #454 for custom templates. Voyager will by default use templates found here:
https://github.com/appscode/voyager/tree/master/hack/docker/voyager/templates

If you want custom template, you mount that as a configmap and pass it as a flag (glob pattern) to voyager operator (--custom-templates).

For example, you can just override defaults template to pass any defaults you like.

@tamalsaha

This comment has been minimized.

Copy link
Member

commented Sep 7, 2017

I have added docs on how to use custom templates. https://github.com/appscode/voyager/blob/master/docs/user-guide/ingress/custom-templates.md

You can try this with appscode/voyager:3.2.0-rc.3 .

tamalsaha added a commit that referenced this pull request Dec 13, 2017
Fix #343 

Supports all valid options for defaults section of HAProxy config from the list from here
https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#4.2-option%20abortonclose

Expects a json encoded map
ie: "ingress.alpha.appscode.com/default-option": {"http-keep-alive": "true", "dontlognull": "true", "clitcpka": "false"}
This will be appended in the defaults section of HAProxy as
```
   option http-keep-alive
   option dontlognull
   no option clitcpka
```

It is upto user to provide valid configurations. Providing invalid options will not run HAProxy. The log will display something like
```
daemon.err: Aug 18 06:14:41 reloader:  [ALERT] 229/061441 (37) : parsing [/etc/haproxy/haproxy.cfg:21]: negation/default is not supported for option 'invalid-options-false'.
daemon.err: Aug 18 06:14:41 reloader: [ALERT] 229/061441 (37) : parsing [/etc/haproxy/haproxy.cfg:22] : unknown option 'invalid-options-true'.
daemon.err: Aug 18 06:14:41 reloader: [ALERT] 229/061441 (37) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
daemon.err: Aug 18 06:14:41 reloader: [ALERT] 229/061441 (37) : Fatal errors found in configuration.
daemon.err: Aug 18 06:14:41 reloader: I0818 06:14:41.308405      18 mount.go:140] failed to run cmd

```
tamalsaha added a commit that referenced this pull request Feb 14, 2018
### Default

```yaml
apiVersion: voyager.appscode.com/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  namespace: default
  annotations:
    ingress.appscode.com/type: NodePort
spec:
  rules:
  - host: voyager.appscode.test
    http:
      port: '8989'
      nodePort: '32666'
      paths:
      - path: /foo
        backend:
          serviceName: test-server
          servicePort: '80'
```
Generated haproxy.cfg:
```
# HAProxy configuration generated by https://github.com/appscode/voyager
# DO NOT EDIT!
global
	daemon
	stats socket /tmp/haproxy
	server-state-file global
	server-state-base /var/state/haproxy/
	# log using a syslog socket
	log /dev/log local0 info
	log /dev/log local0 notice
	tune.ssl.default-dh-param 2048
	ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK
defaults
	log global
	# https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#4.2-option%20abortonclose
	# #403
	option dontlognull
	option http-server-close
	# Timeout values
	timeout client 50s
	timeout client-fin 50s
	timeout connect 50s
	timeout server 50s
	timeout tunnel 50s
	# Configure error files
	# default traffic mode is http
	# mode is overwritten in case of tcp services
	mode http
frontend http-0_0_0_0-8989
	bind *:8989 
	mode http
	option httplog
	option forwardfor
	acl is_proxy_https hdr(X-Forwarded-Proto) https
	acl acl_voyager.appscode.test hdr(host) -i voyager.appscode.test:8989
	acl acl_voyager.appscode.test:foo path_beg /foo
	use_backend test-server.default:80 if acl_voyager.appscode.test acl_voyager.appscode.test:foo
backend test-server.default:80
	server pod-test-server-68ddc845cd-x7dtv 172.17.0.4:8080
```

### With use-node-port annotation

```yaml
apiVersion: voyager.appscode.com/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  namespace: default
  annotations:
    ingress.appscode.com/type: NodePort
    ingress.appscode.com/use-node-port: "true"
spec:
  rules:
  - host: voyager.appscode.test
    http:
      port: '8989'
      nodePort: '32666'
      paths:
      - path: /foo
        backend:
          serviceName: test-server
          servicePort: '80'
```
Generated haproxy.cfg:
```
# HAProxy configuration generated by https://github.com/appscode/voyager
# DO NOT EDIT!
global
	daemon
	stats socket /tmp/haproxy
	server-state-file global
	server-state-base /var/state/haproxy/
	# log using a syslog socket
	log /dev/log local0 info
	log /dev/log local0 notice
	tune.ssl.default-dh-param 2048
	ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK
defaults
	log global
	# https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#4.2-option%20abortonclose
	# #403
	option dontlognull
	option http-server-close
	# Timeout values
	timeout client 50s
	timeout client-fin 50s
	timeout connect 50s
	timeout server 50s
	timeout tunnel 50s
	# Configure error files
	# default traffic mode is http
	# mode is overwritten in case of tcp services
	mode http
frontend http-0_0_0_0-8989
	bind *:8989 
	mode http
	option httplog
	option forwardfor
	acl is_proxy_https hdr(X-Forwarded-Proto) https
	acl acl_voyager.appscode.test hdr(host) -i voyager.appscode.test:32666
	acl acl_voyager.appscode.test:foo path_beg /foo
	use_backend test-server.default:80 if acl_voyager.appscode.test acl_voyager.appscode.test:foo
backend test-server.default:80
	server pod-test-server-68ddc845cd-x7dtv 172.17.0.4:8080
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.