Provide an official provision for tie-breaking events with the same @timestamp

[segevfiner]

Description of the issue:
ECS doesn't currently have an official provision for tie-breaking events with the same @tiemstamp. This means that saved queries in Kibana populated by beats and the Kibana logs app don't have a way of doing so and end up displaying such events in the wrong order.

This is very inconvenient for log events which can often be written quite fast in succession.

ECS should have a documented official provision for this, and the relevant Kibana apps/plugins and beats pre-loaded objects should be modified to use it by default.

This can be a set of fields that are used for this by default, or a new field dedicated for it that beats and apps should generate.

Any additional context or examples:

The Kibana logs app defaults the tie-breaker field to _doc and no longer makes it settable, referring you to the ECS schema, which has no field for this, leaving you between a rock and a hard place.

[webmat]

Can you give a bit more details around the need here? Are you simply looking for an official field you can fill with values depending on the context, to help with tie breaking?

Or are you also looking for a more prescriptive recommendation of how to do tie breaking in these situations?

[segevfiner]

I believe this is something that should be handled by default in the ECS schema, beats and Kibana apps. So that log messages and other events retain their order as much as possible.

As it currently stands, log messages/events that are written with the same time lose their order, and require you to manually do a secondary sort. This is something that is not even possible to do in the Kibana logs app, and the tie-breaking field there is fixed (Cannot be changed in the UI) to the not so useful _doc field, citing the ECS schema. In other words, the Kibana logs app has no useful to keep the logs sorted properly.

Of course, what kind of field this should be, and how it should be filled, will have to be specified, and beats, Kibana and others will have to be modified for it.

[ebeahan]

For data sources that publish such a value, the event.sequence field has a place in this discussion. Of course, this is only a small piece of the full issue described here, but I did want to call it to attention.

The EQL search API in Elasticsearch is one example where event.sequence is used as the default tiebreaker field.

[andrewthad]

event.sequence feels like a good answer here. As @ebeahan mentions, certain data sources (firewalls at least) make this available. If a data source does not provide it, elastic beats could fill it in with a monotonically increasing counter. The counter probably would get reset to zero when the beat restarts, but that's how sequence numbers tend to work with everything else.

[adrian-skybaker]

This is something that is not even possible to do in the Kibana logs app, and the tie-breaking field there is fixed (Cannot be changed in the UI) to the not so useful _doc field

Note that based on my quick experiment to workaround this issue, even if you copy event.sequence to the _doc field (eg in functionbeat), the resulting sort in Kibana -> Discover is mistakenly in ascending order, not descending. So you see events sorted first by timestamp descending, then within the same timestamps in ascending sequence order.

I suspect https://discuss.elastic.co/t/not-able-to-sort-data-in-kibana-ui/254102/11 is reporting this same issue.