Postifx polluted fields

[Callisto88]

Hi,

I noticed that some message ID are messing up with Kibana fields
I have more than 7000 polluted fields like this one

{
    "program": "postfix/cleanup",
    "message": "ADD961003206D: message-id==?UTF-8?B?PDE5?=? =?UTF-8?B?MTIyMDExMjkxNjYxMUBDbGllbnQtMDQ1LnJ1YmluZXR0ZXJpZS5sb2NhbD4=?=",
    "received_at": "2019-12-20T10:29:09.784Z",
    "postfix_queueid": "ADD961003206D",
    "tags": [
      "_grok_postfix_success"
    ],
    "postfix_?UTF-8?B?MTIyMDExMjkxNjYxMUBDbGllbnQtMDQ1LnJ1YmluZXR0ZXJpZS5sb2NhbD4": "?=",
    "ecs": {
      "version": "1.1.0"
    },
    "@version": "1",
    "postfix_message-id": "=?UTF-8?B?PDE5?=?"
  }
}

Does anyone ever experienced such issue ?
Is there any way to sanitize data from message field to avoid this problem ?

Thank you in advance

[whyscream]

I remember there was a similar report in the past, but I was unable to reproduce.
Is it possible that you can supply the original logging from postfix for such a message, based on the queue id and the timestamp? All relevant data is already there, in the message field.

[whyscream]

The problem is of course with the message id containing a space, which is unexpected by the grok pattern. As the message-id is only logged by the cleanup daemon, and it's the only thing that's being logged, it might be possible to restrict whatever we extract from the cleanup log lines. I'll take a look

[Callisto88]

Hi @whyscream, thank you for your answer, i'll do also some tests on my side.

[whyscream]

I tried to create a fix in #143. The message-id line from clean was handled by the logstash kv plugin, but that's impossible when the value of a key-value pair contains spaces. Not very elegant, but I don't see any other way.

@Callisto88 Could you verify that it works?