multiline parsing postfix logs? #7853
-
|
Hi, is it possible with multiline parser to parse logs from postfix mail server and glue them together? Common log output from postfix: All lines with the same ID belongs to the same mail processing. But logs are not ordered. Log lines without an ID are only connection logs. It would be great to get an output like this: {
"time": "0000-10-10T15:59:29+09:00",
"hostname": "mail",
"process": "postfix/smtpd[1827]",
"queue_id": "3D74ADB7400B",
"client_hostname": "example.com",
"client_ip": "127.0.0.1",
"message_id": "f93388828093534f92d85ffe21b2a719@example.info",
"from": "test2@example.info",
"messages": [
{
"time": "0000-10-10T15:59:30+09:00",
"to": "test@example.to",
"status": "sent",
"message": "to=<test@example.to>, relay=example.to[192.168.0.20]:25, delay=1.7, delays=0.02/0/1.7/0.06, dsn=2.0.0, status=sent (250 [Sniper] OK 1539154772 snipe-queue 10549)"
},
{
"time": "0000-10-10T15:59:30+09:00",
"to": "test2@example.to",
"status": "sent",
"message": "to=<test2@example.to>, relay=example.to[192.168.0.20]:25, delay=1.7, delays=0.02/0/1.7/0.06, dsn=2.0.0, status=sent (250 [Sniper] OK 1539154772 snipe-queue 10549)"
}
]
}Source from both: It looks like it's possible with logstash: Is something like this possible with fluent-bit multiline parsing? Thank you |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 3 replies
-
|
I don't think normal multiline parsing can do this for you. The other thing I was thinking is to parse out the ID then use that in a second stage processing, e.g. send all lines for ID x to a specific file named by the ID, then tail that file and reconstruct as a normal multiline. Or use the ID in the tag and get a LUA filter to reconstruct them in the same pipeline maybe? |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the suggestions and ideas! I was unsure if this is possible at all. |
Beta Was this translation helpful? Give feedback.
-
|
It's possible, the question is probably more how and where to do it, e.g. a query engine you're sending your logs to (Loki, Elastic, etc.) may be better placed to do it too as it can scale resources then independently as required. |
Beta Was this translation helpful? Give feedback.
-
|
I send it to Loki with json, thanks to fluentbit. Also filtering is possible in Grafana, thanks to the same ID. But this is limited to every single mail. Also I want to have some real advantages from this tools and not only some (slow) fancy web-ui. For me, investigate specific problems is always faster in the shell, using journalctl with some grep/awk. With Grafana I want to see the "big picture" of the infrastructure.
If the VM where Loki (and Grafana) runs, should also handle all log processing from 100s of other VMs, then they may need to scale. But if proccessing is done on every single VM before the logs and metrics will be sent to Loki, I don't think much scaling is needed. |
Beta Was this translation helpful? Give feedback.
I don't think normal multiline parsing can do this for you.
You may be able to do it with a more powerful LUA filter or stream processing.
Or a custom plugin (e.g. Golang) to handle it.
The other thing I was thinking is to parse out the ID then use that in a second stage processing, e.g. send all lines for ID x to a specific file named by the ID, then tail that file and reconstruct as a normal multiline.
Or use the ID in the tag and get a LUA filter to reconstruct them in the same pipeline maybe?
Or maybe something dirty like running grep via
execon the file to just get matching lines for an ID?