third party logs are not parsed correctly from fluentbit -> fluentd aggregator -> Malcolm

[mmguero]

The tested configuration for getting third party logs to Malcolm has been fluentbit -> Malcolm (see the docs).

Another configuration that should be valid, though, is for fluentbit clients to forward to a fluentd aggregator which then sends to Malcolm.

At the moment, this is not handled correctly. The messages are, I believe, being nested another level deeper which is not picked up by Malcolm's Logstash beats pipeline.

The pieces in play:

• filebeat.env
• filebeat.yaml
• the beats pipeline

Ideally what would happen is that I detect it in the beats pipeline (probably here) and adjust accordingly. or perhaps there's a way in the filebeat config itself in the processors section.

[mmguero]

So I've done some digging: fluentd doesn't have direct TCP output, but (at least this is what the reporter of this issue was using) the loomsystems third-party plugin can be used to forward to our filebeat TCP input listener (the same way fluent-bit could do natively).

I'm attaching a few files that indicate the difference between the output from fluentbit vs. that from fluentd with the loomsystems output plugin.

These files are exported documents from the Dashboards Discover app:

fluentd.json
fluentbit.json

Note the diff:

Looks like an extra message level being added in?

[mmguero]

Another way to look at it, without going through the opensearch document. This is straight from the wire:

fluent-bit

{
  "date": 1704730190.658066,
  "mem": {
    "Mem.total": 65763568,
    "Mem.used": 23649400,
    "Mem.free": 42114168,
    "Swap.total": 15625212,
    "Swap.used": 1362432,
    "Swap.free": 14262780
  },
  "module": "mem",
  "host": {
    "name": "sgrover"
  }
}

fluentd

{
  "timestamp": 1704730229,
  "host": "10.9.0.215",
  "message": {
    "date": 1704730228.658081,
    "mem": {
      "Mem.total": 65763568,
      "Mem.used": 23763928,
      "Mem.free": 41999640,
      "Swap.total": 15625212,
      "Swap.used": 1362432,
      "Swap.free": 14262780
    },
    "module": "mem",
    "host": {
      "name": "sgrover"
    }
  }
}