nginx + Consul Backend

This repository contains an nginx module extension for dynamically choosing a healthy backend by communicating directly with HashiCorp Consul's API.

This code is for example purposes only. It demonstrates both the ability to call Go code from C and the ability to link nginx directly to Consul! It is not production ready and should be considered inspiration only.

Usage

This module installs a consul directive inside the location block, and sets the resulting $backend variable to one of the healthy IP:PORT by the given service.

http {
  server {
    listen       80;
    server_name  example.com;

    location /my-service {
      consul $backend tag-name@service-name@consul-server-ip-port;
      proxy_pass http://$backend;
    }
  }
}

In this example, when a request to "http://example.com/my-service" comes to nginx, nginx invokes the consul directive and looks for a healthy service named "service-name", returning a random entry from the list. Then it utilized nginx's built-in proxy_pass to send traffic to that IP:PORT.

To put it another way, requests to "http://example.com/my-service" are load-balanced among the instances registered in a Consul service, using Consul's health checks to remove unhealthy backends automatically.

Architecture

Instead of re-inventing the wheel, this module uses Consul's existing Golang API client. The majority of the code is written in Go, with small glue pieces to wire it back into the required C code for nginx. This makes use of golang shared C libraries.

Positively, this gains all the benefits of using the official API client library, including configuration via familiar environment variables, connection pooling and multiplexing, and time-tested stability. This saves the need to write and maintain a new client library written in pure C, and allows us to showcase a really cool feature of Go - shared C libraries. Each request goes through Consul, meaning the probability of routing traffic to an unhealthy host is significantly lower than other solutions.

Negatively, this requires Golang to compile the dynamic library .so file. In theory, this could be compiled in advance by a CI/CD system. There is no need for the Golang runtime, since the runtime is compiled into the dynamic library. Additionally, each request goes through Consul. Thus using a local agent is required for performance and latency reasons.

The general flow is as follows:

1. A request comes into nginx that matches a defined location block with a consul directive.

2. nginx calls the ngx_http_consul_backend function with two arguments.

3. The first argument is the variable in which to store the result (e.g. $backend).

4. The second argument is the name of the Consul service to route to (e.g. my-service).

5. The ngx_http_consul_backend calls dlopen on the shared C library (the .so file mentioned above), and executes the Go function by calling its symbol.

6. The Go function communicates with Consul using the official API client library, compiles a list of IP:PORT results, and then chooses a random result to return.

7. The IP:PORT is returned to the ngx_http_consul_backend function, which then sets the result as the defined variable (e.g. $backend).

8. Usually the next step is to use the built-in proxy_pass directive to send traffic to that host.

Installation

This installation guide uses ubuntu/debian. Adapt as-needed for other platforms.

Prerequisites

• Golang >= 1.9
• Standard build tools, including GCC

Steps

1. Install the necessary build tools:

   $ apt-get -yqq install build-essential curl git libpcre3 libpcre3-dev libssl-dev zlib1g-dev

2. Download and extract nginx source:

   $ cd /tmp
   $ curl -sLo nginx.tgz https://nginx.org/download/nginx-1.17.2.tar.gz
   $ tar -xzvf nginx.tgz

3. Download and extract the nginx development kit (ndk):

   $ cd /tmp
   $ curl -sLo ngx_devel_kit-0.3.1.tgz https://github.com/simpl/ngx_devel_kit/archive/v0.3.1.tar.gz
   $ tar -xzvf ngx_devel_kit-0.3.1.tgz

4. Download/clone this repository:

   $ git clone https://github.com/JasonHonor/ngx_http_consul_backend_module.git /go/src/github.com/JasonHonor/ngx_http_consul_backend_module

5. Compile the Go code as a shared C library which nginx will dynamically load. This uses CGO and binds to the nginx development kit:

   $ cd /tmp/ngx_http_consul_backend_module/src
   $ mkdir -p /usr/local/nginx/ext
   $ CGO_CFLAGS="-I /tmp/ngx_devel_kit-0.3.1/src" \
       go build \
         -buildmode=c-shared \
         -o /usr/local/nginx/ext/ngx_http_consul_backend_module.so \
         src/ngx_http_consul_backend_module.go

   This will compile the object file with symbols to /usr/local/nginx/ext/nginx_http_consul_backend_module.so. Note that the name and location of this file is important - it will be dlopened at runtime by nginx.

6. Compile and install nginx with the modules:

   $ cd /tmp/nginx-1.17.2
   $ CFLAGS="-g -O0" \
       ./configure \
         --with-debug \
         --add-module=/tmp/ngx_devel_kit-0.3.1 \
         --add-module=/go/src/github.com/JasonHonor/ngx_http_consul_backend_module
   $ make
   $ make install

7. Add the required nginx configuration and restart nginx:

   http {
     server {
       listen       80;
       server_name  example.com;
   
       location /my-service {
         consul $backend tag-name@service-name@consul-server-ip-port@redirect_path;
         proxy_pass http://$backend;
       }
     }
   }

   Unlike other solutions, you will not have to restart nginx each time a change happens in the Consul services. Instead, because each request delegates to Consul, you will get real-time results, and traffic will never be routed to an unhealthy host!

Development

There is a sample Dockerfile and entrypoint which builds and runs this custom nginx installation with all required modules.

Alternatives

• Consul Template
• nginx upstream sync