Implement additional text/template sorting methods

[johnbuhay]

In reference to the following:
closed #385
open #190

Two methods that would help tremendously:

Reverse sort
Sort map keys and/or values by length

[odedlaz]

@jnbnyc, can you give a usecase for this?

[bacongobbler]

#385 (comment)

[johnbuhay]

@odedlaz @hypergig
Summary:

I am using confd to reconfigure haproxy dynamically whenever a new kubernetes pod registers itself with etcd. I was registering a thousand keys to be able to easily generate the haproxy.cfg, however all those etcd transactions are exhausting system resources.

For the time being my team has increased the TTL (6hrs) so that these transactions are less frequent and in the meantime are figuring out ways to collapse keys. Unfortunately if a pod is removed now, it is still registered for that extremely long time until confd re-configures haproxy.

Usecase:

You'll see that we are using path_beg matching for routes in haproxy because it is the most efficient.
Reverse sorting allows me to generate the acls is longest string first. To do this however, I now need a completely new branch structure to 'range' on and of course, more keys.

Here is the important part of the registration service:
( where branch function creates etcd dirs with TTLs and leaf creates etcd keys )

FIBONACCI=89  # some static number to generate the reverse sort
...
LEN=${#url}
ID=$(($FIBONACCI - $LEN))
CLAIM_PATH="$HAPROXY_ETCD_BACKENDS_PATH/claims/$ID/${url//\//-}"
leaf "$CLAIM_PATH/url" $url
leaf "$CLAIM_PATH/frontend" $MELD_FRONTEND_NAME
leaf "$CLAIM_PATH/backend" $MELD_BACKEND_NAME

leaf "$BACKEND_PATH/claims/${url//\//-}" $url

Here is the gist of it:
https://gist.github.com/jnbnyc/bcc9d3725194b5e6aa90

Let me know if you need clarification on anything.

[johnbuhay]

so thus my haproxy.tmpl is looking like

frontend http-in
    bind *:8080

    {{if exists "/services/domain"}}{{$domain := getv "/services/domain"}}acl host_{{join (split $domain ".") "-"}} hdr(host) -i {{$domain}}{{end}}

    {{range $claims := ls "/services/claims"}}{{range $c := ls (printf "/services/claims/%s/" $claims)}}{{$url := (printf "/services/claims/%s/%s/url" $claims $c | getv)}}{{$frontend := (printf "/services/claims/%s/%s/frontend" $claims $c | getv)}}{{$backend := (printf "/services/claims/%s/%s/backend" $claims $c | getv)}}
    acl {{$frontend}}{{$c}} path_beg -i {{$url}}
    use_backend {{$backend}}{{$c}} if {{$frontend}}{{$c}}
    {{end}}{{end}}
#  end new frontend claims

    default_backend nomatch


{{range $backends := ls "/services/backends"}}
   {{$name := (printf "/services/backends/%s/name" $backends | getv)}}
   {{range $c := ls (printf "/services/backends/%s/claims" $backends)}}{{$url := (printf "/services/backends/%s/claims/%s" $backends $c | getv)}}
backend {{$name}}{{$c}}
    balance leastconn
    option httpchk HEAD {{$url}} HTTP/1.1\r\nHost:company.website.com
    {{range $servers := ls (printf "/services/backends/%s/servers" $backends)}}{{$server := (printf "/services/backends/%s/servers/%s" $backends $servers | getv)}}
    server {{$name}}{{$c}}-{{join (split $server ":") "-"}} {{$server}} check{{end}}
   {{end}}{{end}}

backend all_servers

    option tcp-check {{range $backends := ls "/services/backends"}}{{$name := (printf "/services/backends/%s/name" $backends | getv)}}{{range $servers := ls (printf "/services/backends/%s/servers" $backends)}}{{$server := (printf "/services/backends/%s/servers/%s" $backends $servers | getv)}}
    server {{$name}}-{{join (split $server ":") "-"}} {{$server}} check{{end}}
    {{end}}

#  end new backend all_servers

    default_backend nomatch

backend nomatch
    errorfile 503 /etc/haproxy/errors/nomatch.http

[HeavyHorst]

@jnbnyc #416
would this help ?

[johnbuhay]

I ran into this error trying to test your PR, which means I missed something when building this version of confd. Do you see where I went wrong?

2016-03-28T17:26:26Z 73e7b66b7bab confd[11]: FATAL Unable to process template /etc/confd/templates/example.tmpl, template: example.tmpl:10: function "sortByLength" not defined

Here is the gist of all the configurations needed to test: https://gist.github.com/jnbnyc/9673fd0448b6889156ec

This is after checking out your PR:

FROM golang

WORKDIR /go
ADD ./ ./
RUN go get github.com/BurntSushi/toml &&  \
    go get github.com/kelseyhightower/confd/backends &&  \
    ./build && ./install
RUN mkdir -p /etc/confd/{conf.d,templates}
ADD *.tmpl /etc/confd/templates/
ADD *.toml /etc/confd/conf.d/

[HeavyHorst]

@jnbnyc

for me it works when i use this Dockerfile:

FROM golang

WORKDIR /go
RUN go get github.com/BurntSushi/toml &&  \
    go get github.com/kelseyhightower/confd &&  \
    cd $GOPATH/src/github.com/kelseyhightower/confd && git remote add sort https://github.com/HeavyHorst/confd.git && git pull sort sort && \
    ./build && ./install
RUN rm -rf $GOPATH/bin
RUN mkdir -p /etc/confd/{conf.d,templates}
ADD *.tmpl /etc/confd/templates/
ADD *.toml /etc/confd/conf.d/

Not sure where you went wrong.
Maybe there is another confd binary in $GOPATH/bin ?

[johnbuhay]

That was it ... don't know how it got there.

root@12dfa1590555:/go# find / -type f -name "*confd" | xargs ls -latr 
-rwxr-xr-x 1 root root  21227512 Mar 28 20:47 /go/bin/confd
-rwxr-xr-x 1 root staff 21247672 Mar 28 20:47 /usr/local/bin/confd
-rwxr-xr-x 1 root root  21247672 Mar 28 20:47 /go/src/github.com/kelseyhightower/confd/bin/confd
root@12dfa1590555:/go# echo $PATH
/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

I can confirm this works, I will apply this to a full haproxy configuration test later.
@HeavyHorst Thank you

[johnbuhay]

In case this could help somebody, check out the difference in cpu between setting TTLs on all of your directories and keys VS only on keys

( etcd cpu utilization on 3 node master cluster )