Support fields validation when synthetic source is used

[jsoriano]

Current fields validation is based on _source, but when using synthetic source, the synthesized _source may look different to the original one, what is causing issues like this one when using TSDB.

We should improve fields validation on system tests to support cases like this one, maybe basing it on the fields API.

We have a similar issue with runtime fields, where fields are not in the source.

[mrodm]

When synthetic source is enabled, in my PR for this issue I'm using the fields key returned instead of the _source key to run the different validation in the fields.

But related to that, when synthetic mode is enabled, should the sample_event.json written be the _source or fields key? @jsoriano

[jsoriano]

But related to that, when synthetic mode is enabled, should the sample_event.json written be the _source or fields key? @jsoriano

Good question. I think that the _source is more what the user would expect, but it is going to be missing info, right?

Thinking about this, I think that ideally we should craft a sample event that contains all fields available to the user and looks more like the non-synthetic _sources. To craft this event we could follow these rules:

• Include all fields available in _source or in fields.
• For each field, check the normalization in their definition. If it is a single element or an array with a single element, and its normalization is not array, then include it as a single element. Otherwise, include it as an array.

But not sure about the effort required to implement this. If it is too much, maybe we can just use a mix of _source + fields, as they are. Or use _source when non-synthetic source is used, and fieldswhen synthetic source is used.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)