updated ECS version to 8.3 and added missing mappings to process inde…

[mitodrummer]

…x for k8s fields

Change Summary

This PR bumps the targeted ECS version to 8.3.1.

The following mappings should now be present on the process datastream template:

• orchestrator.resource.ip
• orchestrator.resource.parent.type
• orchestrator.cluster.id
• container.image.hash.all

Release Target

8.4

Q/A

For mapping changes:

• I ran make after making the schema changes, and committed all changes

[mitodrummer]

The mappings seem to appear in the generated output, but when I ran make run-registry, they never got updated on my local ES. Kibana is talking to the local registry server fine, but yea. Not sure if I missed something.

Thanks.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-08-10T16:13:19.605+0000

• Duration: 7 min 5 sec

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[mitodrummer]

/test

[mitodrummer]

Looking further I notice the new fields appear in the "/endpoint-package/out/schema/linux_event_model_event/generated/beats/fields.ecs.yml" file, but not in /endpoint-package/package/endpoint/data_stream/process/fields/fields.yml

[pzl]

cc @ferullo is Endpoint sending macs in this new format now?

[pzl]

can you please update package/endpoint/data_stream/process/sample_event.json with values for the fields added (container.image.hash.all, and the three orchestrator fields)?

[mitodrummer]

@pzl I noticed you bumped version to 8.4.1. Will it be possible to get this change in for the 8.4 release? cc @lrishi

[pzl]

@mitodrummer the endpoint package 8.4.0 hit feature freeze and released already. But we are likely to ship a package 8.4.1 before the stack 8.4.0 release. Users on any 8.4.0 stack can receive higher-numbered package versions (i.e. they will get package 8.4.1)

[lrishi]

@pzl Not having these changes breaks our K8s Dashboard. We assumed that endpoint-package 8.4.0 would have ECS 8.3.x fields, but it doesn't seem to be the case. Is there any way we can get these changes into 8.4.0?
Cc: @norrietaylor @qcorporation

[pzl]

Are there additional fields that were intended to be added in this PR?

The only actual mapping changes I see are the addition of these fields:

[process]

• container.image.hash.all
• orchestrator.cluster.id
• orchestrator.resource.ip
• orchestrator.resource.parent.type

Are these all the ECS 8.3 fields we are talking about that needed to be added here?

The endpoint package 8.4.0 has released and shipped already. There should (presumably) be no problem including these changes in a followup patch release for the package here (the package version would be 8.4.1). Users installing the 8.4.0 kibana and elastic stack will receive the new patch release. New packages can be released out-of-band from the Elastic stack releases, as it is sent to a public repository on the internet that kibana fetches updates for regularly.

For reference for future cycles, we need mapping changes earlier in the release cycle if we want to time them with stack release. The package feature-freeze will soon be at least at the same time as stack feature freeze, if not earlier, so that testing can commence during the freeze period (and earlier)

[lrishi]

Yes, these are ECS 8.3.

[lrishi]

There was an assumption here that 8.n will have all the content from 8.n-1 release. We should've checked it in time, but I think we should also try to keep endpoint package up-to-date with (at-least) n-1 release, otherwise we should get rid of the matching release numbers, they keep causing confusion :)
cc: @ferullo

[pzl]

The endpoint package does not automatically pull in ECS changes for each release. Any fields that need to be added for a given release must have an explicit PR adding them. I don't believe there are plans to have the mappings change each release without specific intent.

For instance, in this case, if we were to update the ECS tag to 8.3.1 automatically to keep in sync, we would not have uncommented the sections in custom_subsets/elastic_endpoint/process/linux_event_model_event.yaml that actually add the fields. So the ECS updates would be limited to description/text changes without a manual step to actually add fields.

[lrishi]

ah, I see. Are you suggesting that we can release EPP 8.4.1 before 8.4.0 stack release to workaround this?

[pzl]

Yes, we will release package 8.4.1 as soon as this PR (and #276) are merged. If there are additional changes to merge or bugs to fix (e.g. perhaps a field typo in this PR) we will cut as many patch releases as needed before the stack 8.4.0 release

[qcorporation]

@lrishi @zizhouW @mitodrummer @norrietaylor

I just had a quick call with @pzl out-of-band to educate myself on timing. Thank you, @pzl, for helping me understand the release cadence between endpoint-package and the stack release.

The endpoint-package release is a less formal process which can be kicked off by @pzl at any time. There's confusion that 8.4.1 package release has any association with 8.4.1 stack release. The major and minor versions of the package release will be associated with the major and minor versions of the stack release. e.g. 8.4.0 stack release can be associated with 8.4.999 of the package release.

What it boils down to is that this change most likely will be included within the next package release, e.g. 8.4.1 and will go out BEFORE the 8.4.0 stack release. @pzl can make this happen. - thank you again.

From my side, there are no concerns here. We will get this change in very soon, and it will get published before our 8.4.0 stack release.

[lrishi]

Awesome, thanks everyone
🙌🏽

[pzl]

👍 🚀