[Fortinet] Could not index event to Elasticsearch. [sentdelta] value out of range

[nicpenning]

I have noticed that once in a while an event won't get ingested because the [sentdelta] field contains a value such as 18446744073706429550 which is larger than the type long required in Elastic. Is there a way to handle bigintegers in Elastic? Otherwise I might just make this a text field or changing the mapping settings to let it index anyways. How has anyone else managed this?

[nicpenning]

Looking further, it seems that this shouldn't be a valid value any ways, so I think something went amiss when the log was shipped or the pipeline mangled it up. From what I can tell, there are no references to [sentdelta] in the pipeline. To be continued.

[enotspe]

Does event.original shows the same value for sentdelta? Is it Fortinet issue or parsing issue? Can you compare it with the log on the firewall itself?

[nicpenning]

I am trying to get it now. Unfortunately this won't log into Elastic with the error and since we have a lot of data in the pipeline it might be hard to isolate. I can try and see if I can find this data in the raw log that is shipped. I don't see anywhere in the pipeline where you would parse this field. I will post what I find.

[nicpenning]

We can push this one back to Fortinet!

This is indeed a bug in the log. That number is huge and is likely incorrect. Will send something there way.

[enotspe]

sentdelta is defined on the template
https://github.com/enotspe/fortinet-2-elasticsearch/blob/master/templates/fortigate-6.2.2

It is not parsed specifically. But kv should handle it.

The value 18446744073706429550 is around 18446.74 Petabytes. I have also received values of duration around 100+ years, so I think it is an issue with Fortigate itself. Probably FortiOS version

[nicpenning]

It works everywhere else. It's just when it has that bugged value, it doesn't match the type long so it doesn't get indexed. Fortinet will need to fix. Otherwise I could add some basic log that if the deltas have 20+ digits then remove that field and tag that log with an error or add a new text field with the value, even though the value is useless.

[enotspe]

sentdelta and rcvddelta only happen on logid=20, which are excluded from all analysis, so you might go ahead and drop completely logid=20 anyway

[nicpenning]

Closing this issue as this event should be dropped or mapped to a string type field depending on an organization's use cases.