Nginx Parser Not Working

[tbickford]

Hi! I am having issues getting Parsers other than the apace parser to function properly. For simplicity purposes I am just trying a simple Nginx Parser but Fluent Bit is not breaking the fields out.

Here is stdout in the Fluent Bit logs for the Nginx Pod:

logging/fluent-bit-grg7s[fluent-bit]: [0] kube.nginx-logs_default_nginx-685a9c0af3f90d0db74c058d7d58316940b6f171d84c52222439e6fca8a2ab8b.log: [1547939295.210102861, {"log"=>"127.0.0.1 - - [19/Jan/2019:23:08:15 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.52.1" "-"
logging/fluent-bit-grg7s[fluent-bit]: ", "stream"=>"stdout", "time"=>"2019-01-19T23:08:15.210102861Z", "kubernetes"=>{"pod_name"=>"nginx-logs", "namespace_name"=>"default", "pod_id"=>"1afad9e7-1b98-11e9-8f79-ce3f301c456d", "labels"=>{"app"=>"nginx-logs"}, "annotations"=>{"fluentbit.io/parser"=>"nginx", "kubectl.kubernetes.io/last-applied-configuration"=>"{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{\"fluentbit.io/parser\":\"nginx\"},\"labels\":{\"app\":\"nginx-logs\"},\"name\":\"nginx-logs\",\"namespace\":\"default\"},\"spec\":{\"containers\":[{\"image\":\"nginx\",\"name\":\"nginx\"}]}}\n"}, "host"=>"node3", "container_name"=>"nginx", "docker_id"=>"685a9c0af3f90d0db74c058d7d58316940b6f171d84c52222439e6fca8a2ab8b"}}]

Here is stdout in the Fluent Bit logs for the Apache Pod (the log is being parsed correctly and the fields are generated):

logging/fluent-bit-grg7s[fluent-bit]: [0] kube.apache_default_apache-d589e52f093f8b65c2fd1397f590134cc4e586a0b4179368d99d0951f6c09cc7.log: [1547939301.000000000, {"log"=>"127.0.0.1 - - [19/Jan/2019:23:08:21 +0000] "GET / HTTP/1.1" 200 45
logging/fluent-bit-grg7s[fluent-bit]: ", "stream"=>"stdout", "time"=>"2019-01-19T23:08:21.302336426Z", "host"=>"127.0.0.1", "user"=>"-", "method"=>"GET", "path"=>"/", "code"=>"200", "size"=>"45", "kubernetes"=>{"pod_name"=>"apache", "namespace_name"=>"default", "pod_id"=>"074645f6-1b97-11e9-8f79-ce3f301c456d", "labels"=>{"app"=>"apache"}, "annotations"=>{"fluentbit.io/parser"=>"apache", "kubectl.kubernetes.io/last-applied-configuration"=>"{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{\"fluentbit.io/parser\":\"apache\"},\"labels\":{\"app\":\"apache\"},\"name\":\"apache\",\"namespace\":\"default\"},\"spec\":{\"containers\":[{\"image\":\"httpd\",\"name\":\"apache\"}]}}\n"}, "host"=>"node3", "container_name"=>"apache", "docker_id"=>"d589e52f093f8b65c2fd1397f590134cc4e586a0b4179368d99d0951f6c09cc7"}}]

As you can see, the Nginx output is not parsing the fields even though the Pod has the Nginx Parser annotated.

Here is the Nginx Pod:

kind: Pod
metadata:
  name: nginx-logs
  labels:
    app: nginx-logs
  annotations:
    fluentbit.io/parser: nginx
spec:
  containers:
  - name: nginx
    image: nginx

Here is the Fluent Bit ConfigMap that I am using:

kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
  labels:
    k8s-app: fluent-bit
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-stdout.conf

  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10

  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc.cluster.local:443
        Merge_Log           On
        K8S-Logging.Parser  On

  output-stdout.conf: |
    [OUTPUT]
        Name        stdout
        Match       kube.*default*
        Logstash_Format On

  parsers.conf: |
    [PARSER]
        Name   apache
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache2
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache_error
        Format regex
        Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$

    [PARSER]
        Name   nginx
        Format regex
        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On
        # Command      |  Decoder | Field | Optional Action
        # =============|==================|=================
        Decode_Field_As   escaped    log

    [PARSER]
        Name        syslog
        Format      regex
        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S

    [PARSER]
        Name java_multiline
        Format regex
        Regex (?<apptime>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}\.\d{3})
        Time_Format %Y-%m-%d %H:%M:%S.%L
        Time_Keep true
        Time_Key apptime

I tested the Nginx log output 127.0.0.1 - - [19/Jan/2019:23:08:15 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.52.1" "-" at https://regex101.com/ with the Regex in the config file and the line did parse correctly so I am not sure what is happening.

I am using Fluent Bit version 1.0.2. I was also experiencing the same issue in lower versions. Any help would be much appreciated! Thank you.

[tbickford]

It looks like the RegEx for Nginx should have been Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")

[esteban1983cl]

This works perfectly

(?<remote_addr>[^ ]) - [(?<proxy_protocol_addr>[^ ])] - (?<remote_user>[^ ]) [(?[^\]])] "(?\S+)(?: +(?[^\"]) +\S)?" (?[^ ]) (?[^ ]) "(?[^\"])" "(?[^\"])" (?<request_length>[^ ]) (?<request_time>[^ ]) [(?<proxy_upstream_name>[^ ])] (?<upstream_addr>[^ ]) (?<upstream_response_length>[^ ]) (?<upstream_response_time>[^ ]) (?<upstream_status>[^ ]) (?<addional_data>[^ ])

source: http://fluentular.herokuapp.com/parse?regexp=%28%3F%3Cremote_addr%3E%5B%5E+%5D*%29+-+%5C%5B%28%3F%3Cproxy_protocol_addr%3E%5B%5E+%5D*%29%5C%5D+-+%28%3F%3Cremote_user%3E%5B%5E+%5D*%29+%5C%5B%28%3F%3Ctime%3E%5B%5E%5C%5D%5D*%29%5C%5D+%22%28%3F%3Cmethod%3E%5CS%2B%29%28%3F%3A+%2B%28%3F%3Crequest%3E%5B%5E%5C%22%5D*%29+%2B%5CS*%29%3F%22+%28%3F%3Ccode%3E%5B%5E+%5D*%29+%28%3F%3Csize%3E%5B%5E+%5D*%29+%22%28%3F%3Creferer%3E%5B%5E%5C%22%5D*%29%22+%22%28%3F%3Cagent%3E%5B%5E%5C%22%5D*%29%22+%28%3F%3Crequest_length%3E%5B%5E+%5D*%29+%28%3F%3Crequest_time%3E%5B%5E+%5D*%29+%5C%5B%28%3F%3Cproxy_upstream_name%3E%5B%5E+%5D*%29%5C%5D+%28%3F%3Cupstream_addr%3E%5B%5E+%5D*%29+%28%3F%3Cupstream_response_length%3E%5B%5E+%5D*%29+%28%3F%3Cupstream_response_time%3E%5B%5E+%5D*%29+%28%3F%3Cupstream_status%3E%5B%5E+%5D*%29&input=192.168.196.96+-+%5B192.168.196.97%5D+-+redflex+%5B10%2FNov%2F2017%3A00%3A12%3A42+%2B0000%5D+%22GET+%2Fapi%2Fv1%2Flogin%2Fstatus+HTTP%2F1.1%22+200+92+%22https%3A%2F%2Fdashboard.rts.onl%2F%22+%22Mozilla%2F5.0+%28Windows+NT+6.1%3B+Win64%3B+x64%29+AppleWebKit%2F537.36+%28KHTML%2C+like+Gecko%29+Chrome%2F61.0.3163.100+Safari%2F537.36%22+550+0.002+%5Bkube-system-kubernetes-dashboard-80%5D+100.112.15.1%3A9090+92+0.002+200&time_format=%25d%2F%25b%2F%25Y%3A%25H%3A%25M%3A%25S+%25z