Adding Datadog config in nginx.conf causes abnormal behavior with "proxy_set_header" directive

[ydokmetas]

Environment:
nginx:1.21.5

When the following code added to nginx.conf, it causes proxy_set_header directive to be ineffective as if the directive does not exist.
If Datadog block removed then the directive works as expected.
Tracing works fine nonetheless, it is just that the header is not set correctly.

load_module modules/ngx_http_datadog_module.so;

http {
  datadog {
    "service": "usage-internal-nginx",
    "sample_rate": 1
  }
  proxy_set_header Host $host;
}

[dgoffredo]

Interesting. I'll attempt to reproduce this next week.

Can you share more of your actual configuration? My first attempt would be to have a location with a proxy_pass and see whether Host (or whatever else I specify in proxy_set_header) is added.

If your config is much different than that, then please share what else you can.

[ydokmetas]

Here is the full config that can be used to reproduce the issue;

user  nginx;
worker_processes  auto;
load_module /usr/lib/nginx/modules/ngx_http_datadog_module.so;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
     datadog {
     "service": "nginx",
     "type": "web",
     "sample_rate": {{ sample_rate | default("1") }},
     "agent_host": "10.244.0.16",
     "environment": "kind-local",
     #"operation_name_override": "handle.request",
     "version": "1.21.5"
     }

    proxy_set_header   Host             $host;
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;
    sendfile        on;
    keepalive_timeout  65;

    server {
    listen       80;
    listen  [::]:80;
    listen 443 ssl;
    server_name   local.dev.com;
    ssl_certificate /etc/nginx/ssl/dev.pem;
    ssl_certificate_key /etc/nginx/ssl/dev.key;

    location / {
        #proxy_set_header Host $host;
        proxy_pass http://my-service:8000;
    }
    }
}

In order to get proxy_set_header working, either datadog block needs to be removed or proxy_set_header Host $host; needs to be moved under location.
`

[dgoffredo]

In my setup, nginx rejects this config:

datadog {
     "service": "nginx",
     "type": "web",
     "sample_rate": {{ sample_rate | default("1") }},
     "agent_host": "10.244.0.16",
     "environment": "kind-local",
     #"operation_name_override": "handle.request",
     "version": "1.21.5"
}

saying

example-nginx-1    | 2022/11/07 21:59:21 [error] 1#1: Error reading "datadog { ... }" configuration block: [json.exception.parse_error.101] parse error at line 23, column 23: syntax error while parsing object key - unexpected '{'; expected string literal
example-nginx-1    | nginx: [error] Error reading "datadog { ... }" configuration block: [json.exception.parse_error.101] parse error at line 23, column 23: syntax error while parsing object key - unexpected '{'; expected string literal
example-nginx-1 exited with code 1

I'm guessing that the {{ sample_rate | default("1") }} syntax goes through some preprocessor first.

Instead, I tried an extended version of your original example, based on example/:

# load_module modules/ngx_http_datadog_module.so;  #  1
                                                   #  2
events {                                           #  3
    worker_connections  1024;                      #  4
}                                                  #  5
                                                   #  6
http {                                             #  7
    access_log /dev/stdout;                        #  8
                                                   #  9
    proxy_set_header Foo Bar;                      # 10
                                                   # 11
    server {                                       # 12
        listen 80;                                 # 13
        # proxy_set_header Boom Boom;              # 14
                                                   # 15
        location /http {                           # 16
            # proxy_set_header Blam Thwack;        # 17
            proxy_pass http://http:8080;           # 18
        }                                          # 19
    }                                              # 20
}                                                  # 21

The proxy_set_header directive has somewhat odd behavior: A context (e.g.
location) inherits the proxy_set_header settings of its ancestors if and
only if the context (location) does not have any of its own proxy_set_header
directives. See the first paragraph of the doc.

nginx-datadog works (as does its predecessor, nginx-opentracing) by
injecting proxy_set_header directives into locations for which tracing is
enabled.

In the example above, the datadog module is not loaded at all. The upstream at
/http will receive the Foo: Bar header because it's set in the http block
and no other proxy_set_header directives are in either the server block or
location block.

If you uncomment line 14, then the upstream will receive the Boom: Boom header
but not the Foo: Bar header.

If you uncomment line 17, then the upstream will receive the Blam: Thwack
header, but neither of the Boom: Boom or Foo: Bar headers.

If you then uncomment line 1, it is as if more proxy_set_header directives
were added to the location (automatically by the module), and so the upstream
will still get the Blam: Thwack header, and a bunch of tracing-related
headers, but still neither of the Boom: Boom or Foo: Bar headers.

I recommend you use proxy_set_header in each location where you want it to
act. It seems to be a common pattern to include a file of such directives
to avoid duplication.

[dgoffredo]

I just got back from a three week vacation, and this issue is not on the top of my TODO list, but I am still aware of this problem and will explore solutions in due time.

Edit: I commented on the wrong issue.