Store Message receive and processing times

[mpfz0r]

With #6022 we've started recording per-message receive and processing timestamps.
We didn't add those timestamps as message fields, because we wanted to make sure every index
has a proper datetime format mapping for those fields.

With a bit of delay 😏 we can finally store those fields on each Message.

Fixes #16284

Refs #16163

Forwarder support: https://github.com/Graylog2/graylog-plugin-enterprise/pull/6560

[mpfz0r]

@Graylog2/architecture

I'm wondering if the timestamp fields are actually that useful.
Wouldn't it be better to just store the processing duration instead?
See
e02943f

[bernd]

@Graylog2/architecture

I'm wondering if the timestamp fields are actually that useful. Wouldn't it be better to just store the processing duration instead? See e02943f

@mpfz0r I think the timestamp fields have their use cases. I agree that the duration is better for dashboards since we can't compute any values in our queries, AFAIK.

We can store the duration as well. Let's call it gl2_processing_duration_ms though.

The new field needs to be added to our index templates. If we don't want to wait again until everyone has the field, we need a migration step that adds the field to all active write indices.

[mpfz0r]

@Graylog2/architecture
I'm wondering if the timestamp fields are actually that useful. Wouldn't it be better to just store the processing duration instead? See e02943f

@mpfz0r I think the timestamp fields have their use cases. I agree that the duration is better for dashboards since we can't compute any values in our queries, AFAIK.

Doing that on the fly would involve a painless scripted query, not ideal, no.

We can store the duration as well. Let's call it gl2_processing_duration_ms though.

Ack 👍

The new field needs to be added to our index templates. If we don't want to wait again until everyone has the field, we need a migration step that adds the field to all active write indices.

Is that necessary? The field will automatically be detected as a numerical value.

[bernd]

Is that necessary? The field will automatically be detected as a numerical value.

@mpfz0r Ah right, it's not a date field like the other two. 👍

[kodjo-anipah]

LGTM processing time, processing duration & receiving time are all there