GELF message is missing mandatory "host" field

[alekseydemidov]

Bug Report

Describe the bug

During getting gelf messages from fluent-bit, i see continuous repeated error in graylog logs:

2020-01-20 21:56:28,029 WARN    [GelfCodec] - GELF message <bla-bla-bla> (received from <10.224.5.83:49072>) is missing mandatory "host" field. - {}

To Reproduce

In GELF documentation: "If you're using Fluent Bit in Kubernetes and you're using Kubernetes Filter Plugin, this plugin adds host value to your log by default, and you don't need to add it by your own."
But looks like that's not true.

That possible to see logs in raw/text format on graylog side and there's not really the 'host' field
I've tried to set Gelf_Host_Key for gelf plugin, but no any intelligible explanations how to use that, any values were ignored

Expected behavior
add neccessary 'host' field into log

Screenshots

Your Environment
Platform: k8s v1.16.4
fluent-bit:1.3.5
graylor:3.1

• Configuration:
  fluent-bit configuration:

  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10

  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off

  output-elasticsearch.conf: |
    [OUTPUT]
        Name            gelf
        Match           kube.*
        Host            log-input-12201.logging.svc.cluster.local
        Port            12201
        Mode            tcp
        Gelf_Short_Message_Key    log

Additional context
This WARN does not prevent logs collection, but overfill journal log of the graylog

[alekseydemidov]

Someone who faces the same issues:
Workaround found:

1. To avoid WARN about 'missing mandatory "host" field.'

  output-elasticsearch.conf: |
    [OUTPUT]
        Name            gelf
        ...
        Gelf_Host_Key             stream
        Gelf_Short_Message_Key    log

2. To avoid ERROR about empty short_message for docker logs like that:
   {"log":"\n","stream":"stdout","time":"2020-01-21T09:12:57.043094018Z"}
   add filter before kubernetes filter:

    [FILTER]
        Name              grep
        Match             kube.*
        Exclude           log ^$

[DarkBlaez]

I can confirm with the fluent-bit 1.5 that this bug exist. Adding the Gelf_Host_key still does not work. The host key is not being passed to Graylog with GELF. Has anyone tested this. Seem if there is a recommendation it should be fully vetted.

DB

[jstewart612]

No this has worked for me. I'm using the latest fluent-bit helm chart from fluent itself and the latest graylog helm chart from stable as of this writing.

[peetasan]

@alekseydemidov when I set Gelf_Host_Key to stream I will get stdout or stderr as source. What can I do if I want something else like container name or similar?

[thiagolinhares]

Same here.
Someone has any updated info about this issue?

[fkoehler]

Any updates on this one? I could not find a way to workaround this properly. The fixes mentioned above only create fixed hosts fields.

[uluzox]

You can omit the Gelf_Host_Key if a variable named host exists. In my case, I use the variable from the Kubernetes Filter Plugin.

[FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        Annotations On
        Labels On

    [FILTER]
        Name nest
        Match *
        Operation lift
        Nested_under kubernetes

I also omit the Gelf_Short_Message_Key by ensuring that the relevant data is stored in a variable called short_message.
For that I use the Modify Filter Plugin

# field version is required in GELF
    # rename field "log" to "short_message" as required by GELF
    [FILTER]
        Name modify
        Match *
        Add version 1.1
        Rename log short_message

Finally my Output plugin looks like this:

[OUTPUT]
        Name gelf
        Match *
        Host somewhere.de
        Port 12202
        Mode tls
        tls On
        tls.verfiy On
        tls.ca_file /ect/ssl/certs/ca-certificates.crt

You can use udp or unencrypted tcp as mode as well (c.f. https://docs.fluentbit.io/manual/pipeline/outputs/gelf#configuration-file-example)

Furthermore https://github.com/fluent/fluent-bit-docs/blob/master/pipeline/outputs/gelf.md#configuration-file-example gives a configuration file example for GELF.

To verify your output. I used the HTTP Plugin instead in order to direct the output to a service that logs the payload e.g. https://hub.docker.com/r/mendhak/http-https-echo, e.g.

[OUTPUT]
    Name  http
    Match *
    Host  192.168.2.3
    Port  80

[DanPhonovation]

Hi,

Could someone clarify a few things for me regarding fluent-bit and GELF support to Graylog?

Lots of tutorials (https://www.xtivia.com/blog/k8s-loggings-graylog-fluent-bit) and (https://vzurczak.wordpress.com/2019/02/13/using-graylog-for-centralized-logs-in-k8s-platforms-and-permissions-management) appear to be using Graylog GELF HTTP input.

Whereas, it appears that fluent-bit expects this input to be the standard GELF TCP input.

Which I tried and started getting logs, but now I get double timestamps in message like so:

2021-05-20T18:55:21.999257639+01:00 stdout F 192.168.1.42 - b2d78eb1-416e-422f-96e6-9924af86fb99 [20/May/2021:17:55:21 +0000] "POST /api/cluster/metrics/multiple HTTP/2.0" 200 808 "https://graylog.internal.net/system/inputs" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0" 3375 0.036 [graylog-phono-graylog-web-graylog] [] 10.244.173.17:9000 820 0.033 200 c6809bb03426c03a4336755fb3b551b2

Note the initial timestamp with my local timezone, followed by regular nginx.

So my questions boil down to 2 I think:

1. Does this GELF plugin send raw TCP or HTTP?

2. If I guessed right with raw TCP, any ideas as to why fluent-bit is adding this info? I checked the source pod logs and they do not contain this info. They start with '192.1681.42 - b2dd78........'

[DanPhonovation]

I solved my problem with "Parser: cri" instead of "docker".

I still however am curious about the tcp/http thing and I'm also still experiencing this bug.

I have a kubernetes_host field. I've tried renaming it. I've tried adding Gelf_Host_Key. Nothing

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[lucardcoder]

any resolution to this ?