[Feat]: Collect (k8s) nginx-ingress logs

[Ismael]

Problem

I want to use netdata's web logs collector to analyze my nginx-ingress logs.
Ideally, I want to find per-host bandwidth usage.

Description

Nginx-ingress only logs to stdout. ND wants a file.
Ideally ND child would mount the node's logs and find /var/log/containers/ingress-nginx-controller*

Importance

really want

Value proposition

1. Analyze nginx-ingress logs with existing collector
2. Potentially enable to analyze all kind of pod logs with custom integrations

Proposed implementation

Add a /var/log/container mount to ND child. Enable web log collector to search for log files using wildcards?

[Ismael]

This is as close as I got. Turns out web log collector already allows for wildcards.

Nginx ingress helm values:

USER-SUPPLIED VALUES:
controller: 
  config:
    log-format-upstream: "$host $remote_addr - \"$request\" $status $bytes_sent \"$http_referer\" \"$http_user_agent\" $request_length $request_time $upstream_response_time $upstream_addr"

Netdata helm values

child:
  extraVolumes:
    - name: containerlogs
      hostPath:
        path: /var/log/pods
        type: Directory
  extraVolumeMounts:
    - name: containerlogs
      mountPath: /var/log/pods
      readOnly: true

  configs:
    go.d-web_log:
      enabled: true
      path: /etc/netdata/go.d/web_log.conf
      data: |
        jobs:
        - name: nginx-ingress
          path: /var/log/pods/*ingress-nginx-controller*/controller/*.log
          log_type: regexp
          regexp_config:
            pattern: (?P<time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{9}-\d{2}:\d{2}) stdout F (?P<host>[^ ]+) (?P<remote_addr>[^ ]+) - "(?P<request>[^"]+)" (?P<status>\d+) (?P<bytes_sent>\d+) "(?P<http_referer>[^"]+)" "(?P<http_user_agent>[^"]+)" (?P<request_length>\d+) (?P<request_time>[\d\.]+) (?P<upstream_response_time>[\d\.]+) (?P<upstream_addr>[^ ]+)

I tried using the CSV log_type but I couldn't get it to match. The error message just said that it couldn't match the line.

For reference, this is a line
'2024-03-24T13:17:51.265395971-03:00 stdout F netdata.test 10.42.0.76 - "GET /api/v2/ HTTP/1.1" 200 4039 "http://netdata.test/spaces/netdata-parent/rooms/local/nodes/cabc8b98-b969-11ee-9a5d-52540049cf1a" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36" 1008 0.004 0.004 10.42.0.40:19999'

And this was my pattern:
$time_local stdout F $host $remote_addr - "$request" $status $bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time $upstream_response_time $upstream_addr

The docs for web_log are pretty confusing. The "jobs" are defined here: https://learn.netdata.cloud/docs/developer-and-contributor-corner/monitor-nginx-or-apache-web-server-log-files-with-netdata , but the configuration here https://learn.netdata.cloud/docs/collecting-metrics/web-servers-and-web-proxies/web-server-log-files#configuration doesn't mention that "jobs" anywhere.

Remaining issue is that I'd like to have bandwidth usage per vhost. Seems this can only be done if I create url_patterns for all my vhosts?