Add OpenTelemetry distributed tracing support via ngx_otel_module module

[itskingori]

The module supports W3C context propagation and OTLP/gRPC export protocol. See updated example and tests below. To note, when enabled, the following span attributes are added automatically:

• http.method
• http.target
• http.route
• http.scheme
• http.flavor
• http.user_agent
• http.request_content_length
• http.response_content_length
• http.status_code
• net.host.name
• net.host.port
• net.sock.peer.addr
• net.sock.peer.port

Testing

Build and start up the services ...

$ docker-nginx-proxy git:(opentelemetry_nginx_module) ✗ docker-compose build
[+] Building 0.9s (15/15) FINISHED                                                                                                                           docker:desktop-linux
 => [proxy internal] load build definition from Dockerfile                                                                                                                   0.0s
 => => transferring dockerfile: 2.40kB                                                                                                                                       0.0s
 => [proxy internal] load .dockerignore                                                                                                                                      0.0s
 => => transferring context: 136B                                                                                                                                            0.0s
 => [proxy internal] load metadata for docker.io/zappi/nginx:1.25.3                                                                                                          0.8s
 => [proxy internal] load build context                                                                                                                                      0.0s
 => => transferring context: 670B                                                                                                                                            0.0s
 => [proxy builder 1/6] FROM docker.io/zappi/nginx:1.25.3@sha256:cc2182d1db65e056cd873b8e170b86516fe39623009a0567bca4fc3f69368321                                            0.0s
 => CACHED [proxy stage-1 2/5] RUN apt-get update -y &&     apt-get install --no-install-recommends -y         ca-certificates         gnupg2         wget &&     rm -rf /v  0.0s
 => CACHED [proxy stage-1 3/5] RUN apt-get update -y &&     apt-get install --no-install-recommends -y       nginx-module-otel &&     rm -rf /var/lib/apt/lists/* /tmp/*     0.0s
 => CACHED [proxy builder 2/6] RUN apt-get update -y &&     apt-get install --no-install-recommends -y       build-essential       ca-certificates       libpcre3       lib  0.0s
 => CACHED [proxy builder 3/6] WORKDIR /usr/src/                                                                                                                             0.0s
 => CACHED [proxy builder 4/6] RUN wget "http://nginx.org/download/nginx-1.25.3.tar.gz" &&     echo "64c5b975ca287939e828303fa857d22f142b251f17808dfe41733512d9cded86 *ngin  0.0s
 => CACHED [proxy builder 5/6] RUN wget "https://github.com/openresty/headers-more-nginx-module/archive/v0.37.tar.gz" &&     echo "cf6e169d6b350c06d0c730b0eaf4973394026ad4  0.0s
 => CACHED [proxy builder 6/6] RUN cd nginx &&     ./configure --with-compat --add-dynamic-module=/usr/src/headers-more &&     make modules                                  0.0s
 => CACHED [proxy stage-1 4/5] COPY --from=builder /usr/src/nginx/objs/*_module.so /etc/nginx/modules/                                                                       0.0s
 => CACHED [proxy stage-1 5/5] COPY ./config/ /etc/nginx/                                                                                                                    0.0s
 => [proxy] exporting to image                                                                                                                                               0.0s
 => => exporting layers                                                                                                                                                      0.0s
 => => writing image sha256:60131fce20fbb00428cbe096c63c1d0e3bdefbd1680121f7d7c008e1f22b1b28                                                                                 0.0s
 => => naming to docker.io/library/nginx-proxy:latest

$ docker-compose up -d
[+] Running 4/0
 ✔ Container docker-nginx-proxy-jaeger-1          Running                                                                                                                                             0.0s 
 ✔ Container docker-nginx-proxy-app-1             Running                                                                                                                                             0.0s 
 ✔ Container docker-nginx-proxy-otel-collector-1  Running                                                                                                                                             0.0s 
 ✔ Container docker-nginx-proxy-proxy-1           Running                                                                                                                                             0.0s

$ docker-compose ps   
NAME                                  IMAGE                                         COMMAND                  SERVICE          CREATED          STATUS          PORTS
docker-nginx-proxy-app-1              caddy:2.1.1-alpine                            "caddy run --config …"   app              48 seconds ago   Up 47 seconds   80/tcp, 443/tcp, 2019/tcp
docker-nginx-proxy-jaeger-1           jaegertracing/all-in-one:1.53                 "/go/bin/all-in-one-…"   jaeger           48 seconds ago   Up 47 seconds   4317-4318/tcp, 5775/udp, 5778/tcp, 9411/tcp, 14250/tcp, 14268/tcp, 6831-6832/udp, 0.0.0.0:16686->16686/tcp
docker-nginx-proxy-otel-collector-1   otel/opentelemetry-collector-contrib:0.92.0   "/otelcol-contrib --…"   otel-collector   48 seconds ago   Up 47 seconds   4317/tcp, 55678-55679/tcp
docker-nginx-proxy-proxy-1            nginx-proxy:latest                            "/docker-entrypoint.…"   proxy            48 seconds ago   Up 47 seconds   80/tcp, 0.0.0.0:8080->8080/tcp

Make a request ...

$ curl -I http://127.0.0.1:8080
HTTP/1.1 200 OK
Date: Fri, 19 Jan 2024 08:53:04 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 12226
Connection: keep-alive
Vary: Accept-Encoding
Accept-Ranges: bytes
Etag: "qlgk6n9fm"
Last-Modified: Thu, 17 Dec 2020 00:32:47 GMT
X-Robots-Tag: "noindex, nofollow"
X-Proxy-Backend-Connect-Time: 0.020
X-Proxy-Backend-Header-Time: 0.038
X-Proxy-Backend-Response-Time: -
X-Proxy-Request-Time: 0.037
X-Trace-ID: e609a408b1ba26ae4226da92313b8116

Inspect the logs ...

$ docker-compose logs proxy --no-log-prefix | jq
{
  "body_bytes_sent": "0",
  "host": "127.0.0.1",
  "http_connection": "",
  "http_referer": "",
  "http_upgrade": "",
  "http_user_agent": "curl/8.5.0",
  "http_x_amzn_trace_id": "",
  "http_x_forwarded_for": "",
  "http_x_proxy_backend_connect_time": "0.020",
  "http_x_proxy_backend_header_time": "0.038",
  "http_x_proxy_backend_response_time": "0.038",
  "otel_trace_id": "e609a408b1ba26ae4226da92313b8116",
  "proxy_connection": "",
  "proxy_x_forwarded_port": "8080",
  "proxy_x_forwarded_proto": "http",
  "proxy_x_forwarded_ssl": "off",
  "proxy_x_request_id": "64c2fb015a1204c5b153da057003a642",
  "remote_addr": "192.168.65.1",
  "remote_user": "",
  "request": "HEAD / HTTP/1.1",
  "request_length": "78",
  "request_time": "0.037",
  "status": "200",
  "time_iso8601": "2024-01-19T08:53:04+00:00"
}

$ docker-compose logs otel-collector
otel-collector-1  | 2024-01-19T09:00:39.349Z    info    service@v0.92.0/telemetry.go:86 Setting up own telemetry...
otel-collector-1  | 2024-01-19T09:00:39.349Z    info    service@v0.92.0/telemetry.go:159        Serving metrics {"address": ":8888", "level": "Basic"}
otel-collector-1  | 2024-01-19T09:00:39.349Z    info    exporter@v0.92.0/exporter.go:275        Development component. May change in the future.        {"kind": "exporter", "data_type": "traces", "name": "debug"}
otel-collector-1  | 2024-01-19T09:00:39.350Z    info    service@v0.92.0/service.go:151  Starting otelcol-contrib...     {"Version": "0.92.0", "NumCPU": 10}
otel-collector-1  | 2024-01-19T09:00:39.350Z    info    extensions/extensions.go:34     Starting extensions...
otel-collector-1  | 2024-01-19T09:00:39.351Z    warn    internal@v0.92.0/warning.go:40  Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks   {"kind": "receiver", "name": "otlp", "data_type": "traces", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
otel-collector-1  | 2024-01-19T09:00:39.351Z    info    otlpreceiver@v0.92.0/otlp.go:102        Starting GRPC server    {"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "0.0.0.0:4317"}
otel-collector-1  | 2024-01-19T09:00:39.351Z    warn    internal@v0.92.0/warning.go:40  Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks   {"kind": "receiver", "name": "otlp", "data_type": "traces", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
otel-collector-1  | 2024-01-19T09:00:39.351Z    info    otlpreceiver@v0.92.0/otlp.go:152        Starting HTTP server    {"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "0.0.0.0:4318"}
otel-collector-1  | 2024-01-19T09:00:39.351Z    info    service@v0.92.0/service.go:177  Everything is ready. Begin running and processing data.
otel-collector-1  | 2024-01-19T09:01:19.771Z    info    TracesExporter  {"kind": "exporter", "data_type": "traces", "name": "debug", "resource spans": 1, "spans": 1}

Trace should now be available in Jaeger ...

References

• Nginx
  • https://nginx.org/en/docs/ngx_otel_module.html
  • https://docs.nginx.com/nginx/admin-guide/dynamic-modules/opentelemetry/
  • https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-open-source/
  • https://github.com/nginxinc/nginx-otel
• OpenTelemetry
  • https://opentelemetry.io/docs/specs/otel/trace/sdk_exporters/jaeger/
• OpenTelemetry C++
  • https://github.com/open-telemetry/opentelemetry-cpp/tree/v1.13.0
  • https://github.com/open-telemetry/opentelemetry-cpp-contrib/tree/webserver/v1.0.4/instrumentation/nginx
• OpenTelemetry Collector
  • https://github.com/open-telemetry/opentelemetry-collector
  • https://hub.docker.com/r/otel/opentelemetry-collector-contrib
  • https://stackoverflow.com/questions/77475771/error-when-running-otel-collector-with-jaeger-in-docker-containers
• Jaeger
  • https://www.jaegertracing.io/docs/1.6/getting-started/
  • https://hub.docker.com/r/jaegertracing/all-in-one

[itskingori]

This cause the collector to log. I used it to confirm the proxy to collector connection before I set up Jaeger to view the trace.

[itskingori]

These don't have to be here. Some like otel_trace & otel_trace_context can be located elsewhere i.e. in the location block.

[itskingori]

The advantage of having a proxy everywhere is we can set headers like this. Figured it would be nice to have this available in the response (which also means it's in the browser too)!

[itskingori]

These are the defaults.

[hadrianvalentine]

I've skimmed. I'll need to come back and read documentation more closely when I have time, but it looks okay 👍