Documentation of dynamic filenames as a workaround for #64 #88

Closed
bryanlarsen opened this Issue Nov 24, 2014 · 12 comments

Projects

None yet

2 participants

I think I've got my workaround for bug #64 and the limitation that filenames cannot be dynamic. I should probably turn this into a documentation PR once things look good. I'm going to write it in that style. My enhancement request is at the bottom.


Examples

Multiple Nginx config files

Nginx is a popular web server, also popular as a proxy. For this example, we are going to create multiple nginx site files, where the site name is listed in a consul key namespace.

For this example to work, you must have your consul-template configured to read it's configuration from a directory rather than a file.

Our strategy is to create a two layer template. Our template generates templates. It is possible to do this by using consul-template for both the outer and inner templates: you use the printf function to write out lines containing {{}}, but this becomes very awkward quite quickly. So instead for this example we'll use a simple sed script to generate the outer layer.

webapp-nginx.conf.outer-template:

upstream %SERVICE% {
{{range service "%SERVICE%"}}
  server {{ .Address }}:{{ .Port }};
{{end}}
}
server {
  listen 80;
  server_name {{key "%SERVICE%/cname"}};
  location / {
    proxy_pass http://%SERVICE%;
  }
}

We'll use consul-template to dynamically generate and trigger a script that will run sed on this file multiple times, once for each site.

Note: in this example, all file paths are elided for simplicity.

webapp-nginx.sh.template:

#!/bin/sh

{{ range ls "webapp" }}
sed -e "s!%SERVICE%!{{ .Key }}!g" -e "s!%ID%!{{ .Value }}!g" < webapp-nginx.conf.outer-template > "{{ .Key }}.conf.template"
{{ end }}

We also use consul-template to dynamically generate a config file for consul-template to process the new templates.

webapp-nginx.consul-template.cfg.template:

{{ range ls "webapp" }}
template {
  source = "{{ .Key }}.conf.templ"
  destination = "{{ .Key }}.conf"
  command = "service nginx reload"
}
{{ end }}

And finally the outer configuration file:

webapp-nginx.cfg:

template {
  source = "webapp-nginx.sh.tmplate"
  destination = "webapp-nginx.sh"
  command = "sh webapp-nginx.sh"
}

template {
  source = "/opt/plugins/webapp-nginx/webapp-nginx.consul-template.cfg.tmpl"
  destination = "/opt/consul-template/config/webapp-nginx.gen.cfg"
}

The big thing missing from the above example that I have in my system is the line service consul-template restart at the bottom of webapp-nginx.sh.template. That's there to force consul-template to reload it's configuration. But you'll notice that there are actually 2 places that should trigger a reload: the execution of webapp-nginx.sh and the writing of webapp-nginx.gen.cfg. Right now I assume that webapp-nginx.gen.cfg finishes first. A fairly safe assumption, but a nasty race if it doesn't.

If consul-template supported the HUP signal to safely reload it's configuration, I could send HUP in both places and it wouldn't matter which finished first.

Owner

Hi @bryanlarsen

Thank you for the example. While it's technically correct, I'm very hesitant to add something like this to the README. I'm fearful that a new user (who does not have such a use case) would be intimidated by the complexity. I would much rather leave this as an issue that search engines can index and we can point people to. What do you think?

I've also separated SIGHUP stuff into #90.

@sethvargo sethvargo changed the title from reload configuration on HUP (also documentation of dynamic filenames and a workaround for #64) to Documentation of dynamic filenames as a workaround for #64 Nov 24, 2014

Agreed that this is too complex for the README. I'd like to see a link to it in the README, though. This example won't magically become unnecessary if you fix #64. It would if you fixed #64 and supported wildcard filenames, though. I haven't opened an issue request for wildcard filenames, because I have no idea how you'd map the input filename to an output filename.

The other nice thing to add would be a discussion on how to gracefully deal with the consul health-check removing the service. That's way too complex for a README.

Owner

@bryanlarsen it's very unlikely we will support wildcard filenames, but #64 is definitely worth adding nonetheless.

I'm not sure what you mean by "gracefully deal with the consul health-check removing the service".

@sethvargo, in my example, if the service disappears it results in a bad config. It can be fixed by using the any argument to the range service call, but that leaves nginx pointing at a bad server. Nicer would be a config that gracefully fails over to a maintenance page if all services fail the health check. That really shows off the power of consul & consul-template. That would be a separate advanced example, it would over-complicate this already complicated example.

@sethvargo: note that this example could be improved if #77 / #33 were implemented. Then the example would do a range on all services with the 'webapp' tag instead of ranging on the webapp key. That's better both because it's simpler to use and because of it's interaction with consul health checks.

Owner

@bryanlarsen (psuedocode), but why couldn't you do something like:

{{if eq len(service "...") 0}}
  # Write the maintenance page
{{else}}
  # Write the good config
{{end}}

Disclaimer: I haven't actually tested that code, so the syntax might be slightly off. But you get the general idea.

@sethvargo, which comment of mine are your replying to? If you're replying to yesterday's comment, I have figured out how to do this, but I haven't written it up yet. It only needs the "good config" part of your if/else clause. If the service isn't available, then the right behaviour is to write out nothing and fall through to the 'default' server clause.

Because no alternate clause is needed, today's comment is saying that {{ range all_healthy_services_with_tag "webapp" }} would eliminate the need for the if clause entirely, as well as eliminating the need for the services to register a key as well as a service.

Owner

@bryanlarsen but:

{{range all_healthy_services_with_tag "webapp"}}

is already a thing... it's:

{{range service "webapp"}}

That's why I'm confused. The default behavior for the service lookup is for passing health...

That doesn't work for me. I want all services with the tag "webapp". Your example returns all services with the name "webapp".

Owner

@bryanlarsen and that's where the "upstream bug" comes from. Consul doesn't currently support querying by a tag to the best of my knowledge.

I've figured out a better way to do this. Rather than using a shell script and config files, I inline both the script and the configuration into upstart init scripts. That fixes both the potential race condition and the necessity for #90. It's still not pretty, but it works.

Multiple Nginx config files

Nginx is a popular web server, proxy and load balancer. For this example, we are going to create multiple nginx site files, where the site name is listed in a consul key namespace. Each site is backed by one or more services with the same name. Nginx balancing is used if there's more than one service with the name.

For this example to work, you must have your consul-template configured to read it's configuration from a directory rather than a file.

Our strategy is to create a two layer template. Our template generates templates. It is possible to do this by using consul-template for both the outer and inner templates: you use the printf function to write out lines containing {{}}, but this becomes very awkward quite quickly. So instead for this example we'll use a simple sed script to generate the outer layer.

webapp-nginx.conf.outer-ctmpl:

upstream %SERVICE% {
{{range service "%SERVICE%"}}
  server {{ .Address }}:{{ .Port }};
{{end}}
}
server {
  listen 80;
  server_name {{key "%SERVICE%/cname"}};
  location / {
    proxy_pass http://%SERVICE%;
  }
}

We then control this with two upstart init scripts. The outer script generates the inner script.

/etc/init/consul-template-webapp-nginx-0.conf

description "Consul Template webapp-nginx plugin outer template"
start on (local-filesystems and started consul)
stop on runlevel [06]
respawn

exec /usr/local/bin/consul-template -template "/tmp/consul-template-webapp-nginx-1.upstart.ctmpl:/etc/init/consul-template-webapp-nginx-1.conf:service consul-template-webapp-nginx-1 restart"

The inner script generates the template files via sed in a pre-start script, and then runs consul-template.

/tmp/consul-template-webapp-nginx-1.upstart.ctmpl

description "Consul Template webapp-nginx plugin inner template"
start on (local-filesystems and started consul)
stop on runlevel [06]
respawn

pre-start script
{{ range ls "webapp" }}
sed -e "s!%SERVICE%!{{ .Key }}!g" -e "s!%ID%!{{ .Value }}!g" -e "s!%HOSTNAME%!$(hostname)!g" < "/tmp/webapp-nginx.outer-ctmpl" > "/tmp/{{ .Key }}.nginx.ctmpl"
{{ end }}
end script

exec /usr/local/bin/consul-template {{ range ls "webapp" }} -template "/tmp/{{ .Key }}.nginx.ctmpl:/etc/nginx/sites-enabled/{{ .Key }}.conf:service nginx reload" {{ end }}

N.B: replace /tmp with something more appropriate.

Owner

Hi @bryanlarsen

I'm going to start work on #64 and that will be part of the new Consul Template release (probably early next year). As a result, I'm going to close out this issue since it will no longer be required. Thank you for all the documentation, and I am glad you were able to get everything figured out.

@sethvargo sethvargo closed this Dec 16, 2014
@darron darron referenced this issue in octohost/octohost Dec 27, 2014
Closed

consul-template instead of octoconfig? #75

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment