Libbeat: Do not overwrite agent.*, ecs.version, and host.name

[cwurm]

Addresses part of #13920 (comment).

Libbeat currently sets a few fields in every event, with no option to turn it off, or to at least not overwrite existing values.

This is a problem when receiving forwarded events (see #13920 for details). The field I'm most concerned about is host.name which is used by the Kibana SIEM app to identify hosts. This PR changes Libbeat to not overwrite host.name and a few other fields when they are already set (see list of fields below). This is technically a breaking change, though I think in almost all cases the current behavior is not what users would expect, and it is creating problems in the wild (eg #13706, discuss #1, discuss #2) - so I would like to make it in 7.x.

A bit more detail on the implementation: Adds a new function MapStr. DeepUpdateNoOverwrite alongside the existing MapStr.DeepUpdate and an overwrite parameter to the add_fields processor (but does not expose it).

The affected fields that will no longer be overwritten if they already exist are:

1. agent.ephemeral_id
2. agent.hostname
3. agent.id
4. agent.type
5. agent.version
6. ecs.version
7. host.name

If we do not want to change the behavior for all these fields we could also refactor the code to only not overwrite host.name - but I think it makes sense to not overwrite any of these fields.

[kvch]

I have a suggestion to eliminate the cryptic true or false as a parameter to DeepUpdate and keep the API of MapStr a "cleaner". This alternative implementation would also reduce the changes required in the codebase.

// add a new private function which handles updating values
// and makes a decision regarding overwriting values
func (m MapStr) deepUpdateMap(d MapStr, overwrite bool) {
    // body of the current DeepUpdate
}

func (m MapStr) DeepUpdate(d MapStr) {
    m.deepUpdateMap(d, true)
}

func (m MapStr) DeepUpdateWithoutOverwrite(d MapStr) {
    m.deepUpdateMap(d, false)
}

[urso]

+1 on not introducing booleans as arguments to select behavior in public APIs.

If we want to introduce some indicator in the public API it should be an enum so it is clear what will happen at the call side e.g. DeepUpdate(m, common.NoOverwrite). Given DeepUpdate is a quite commonly used API (potentially by community beats) I'd prefer to keep it stable => new function/method. Alternative name to DeepUpdateWithoutOverwrite might be DeepUpdateNoOverwrites.

[cwurm]

@kvch @urso Makes sense, I've changed to a new DeepUpdateNoOverwrite function. Let me know if that works.

[kvch]

LGTM. Please update the section in the PR description about DeepUpdate.

[cwurm]

@kvch Thanks, done. Do you mind formally approving the PR?