[filebeat][o365] Mapping problem on o365.audit.AdditionalInfo

[Daniel314]

Hi,

I've observed that the O365 module in FileBeat sometimes treats o365.audit.AdditionalInfo as a JSON string, and sometimes as an object with some sub-fields instead. Why do I say this? In Kibana, it will report this field as having a type conflict, and when I look at the text version of this field it's a JSON string. The non-text version of this field has sub-fields. I've seen this happen with FileBeat version 7.17(.7?) and with version 8.12.0.

I did post a note about this on the Elastic discussion forum, but never received any response about it.

I'm running Filebeat on Ubuntu 22.04. I don't push the O365 logs directly to Elasticsearch -- I push them to Logstash using a Beats input, and from there push to Elasticsearch. I've worked around this issue by having Logstash check to see if this field is a string, and convert it to an object if it is a JSON string,

Thanks,

  - Daniel

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[efd6]

Possibly related to #37800.

@Daniel314 Do you have example events that have been rejected. I have a PR that addresses an issue like this for another field, so if we have additional test cases that we can add that would be helpful.

The related issue linked above also uses logstash and I have a suspicion that the issue may lie there. Are you prepared to share non-sensitive parts of the logstash pipeline?

[Daniel314]

@efd6 Sorry -- I never collected any samples of the rejects.
I didn't go beyond what is with the initial case creation. My Logstash config is pretty straightforward: Logstash beats listener -> Kafka -> Logstash minimal processing (add several fields, force datatypes on other fields) -> Elastic.

After I noticed the issue, I added a ruby filter to my Logstash pipeline to check if the [o365][audit][AdditionalInfo] field is a string and treat it as a JSON string, convert that to structured data, then store in the same field in that situation. If there are any particular types of O365 log events that you would like me to catch and post here, please let me know what you would like to see (they will be what I pull out of Elastic after the fact, unless you have something specific that you're after).

I've attached my Logstash configs (mostly intact).
LogstashProcess.txt
LogstashInput.txt

[Daniel314]

@efd6 After looking at your possibly related case (37800), I went back and found indexes where o365.audit.AdditionalInfo was set to a string.

In every case, the string for this field was the same: "{"environmentName":"Default-b31b408e-0c8a-49cf-aef8-27b324d5b8b6","actionName":"PUT/PROVIDERS/MICROSOFT.POWERAPPS/APIS/CONNECTIONS/.User"}"

After implementing the Logstash config in my previous post to treat o365.audit.AdditionalInfo as a JSON string and convert it, I am seeing o365.audit.AdditionalInfo.actionName as a string field in my newer indices, with "PUT/PROVIDERS/MICROSOFT.POWERAPPS/APIS/CONNECTIONS/.User" as its value.

[Daniel314]

I performed another check for a captured ZWSP with the following command:
curl -sS -X GET 'http://localhost:9200/MYINDEX/_doc/T_B_0YwB6LoeI1AplCp-' | hexdump -C

The relevant lines in the output are:

00000760  2c 22 41 64 64 69 74 69  6f 6e 61 6c 49 6e 66 6f  |,"AdditionalInfo|
00000770  22 3a 22 7b 5c 22 65 6e  76 69 72 6f 6e 6d 65 6e  |":"{\"environmen|
00000780  74 4e 61 6d 65 5c 22 3a  5c 22 44 65 66 61 75 6c  |tName\":\"Defaul|
00000790  74 2d 62 33 31 62 34 30  38 65 2d 30 63 38 61 2d  |t-b31b408e-0c8a-|
000007a0  34 39 63 66 2d 61 65 66  38 2d 32 37 62 33 32 34  |49cf-aef8-27b324|
000007b0  64 35 62 38 62 36 5c 22  2c 5c 22 61 63 74 69 6f  |d5b8b6\",\"actio|
000007c0  6e 4e 61 6d 65 5c 22 3a  5c 22 50 55 54 2f 50 52  |nName\":\"PUT/PR|
000007d0  4f 56 49 44 45 52 53 2f  4d 49 43 52 4f 53 4f 46  |OVIDERS/MICROSOF|
000007e0  54 2e 50 4f 57 45 52 41  50 50 53 2f 41 50 49 53  |T.POWERAPPS/APIS|
000007f0  2f 43 4f 4e 4e 45 43 54  49 4f 4e 53 2f 2e 55 73  |/CONNECTIONS/.Us|
00000800  65 72 5c 22 7d 22 2c 22  57 6f 72 6b 6c 6f 61 64  |er\"}","Workload|

I'm not seeing any ZWSP characters. This entry was generated via Filebeat 7.17.6, before I started checking to see if this field was a string and explicitly converting it to a structure. Hopefully this helps....

[efd6]

Thanks @Daniel314 those details are very helpful.