RFC Stage 1: TSDB Dimensions

[agithomas]

Have you signed the contributor license agreement? yes
Have you followed the contributor guidelines? yes
For proposing substantial changes or additions to the schema, have you reviewed the [RFC process] (https://github.com/elastic/ecs/blob/main/rfcs/README.md)? yes
If submitting code/script changes, have you verified all tests pass locally using make test?
If submitting schema/fields updates, have you generated new artifacts by running make and committed those changes?
Is your pull request against main? Unless there is a good reason otherwise, we prefer pull requests against main and will backport as needed.
Have you added an entry to the CHANGELOG.next.md?

[agithomas]

From Stage 0 to Stage 1, fields chosen as dimension fields were re-assessed here.

Stage 1 thus has

Three additions : cloud.account.id, cloud.region, cloud.availability_zone and
One deletion : cloud.project.id

in comparison with what is initially estimated in Stage 0.

No similar changes are expected in near future.

[agithomas]

Related Stage 0 RFC

[ruflin]

Overall LGTM. @lalit-satapathy and team, please dive into the details of each field.

One general comment: There is currently a merger ongoing with semconv and in otel everything is a dimension which currently does not work with Elasticsearch. Before a lengthy discussion pops up around should this be in ECS or not, I would like to side step this and keep this PR moving.

There needs to be a more general discussion to have how we store additional info specific to Elasticsearch related to ECS in the future merged schema (or besides it). Lets do it in a separate issue.

[agithomas]

Reasoning behind inclusions of fields as dimensions and changes from Stage 0 to Stage 1 are mentioned below

Field name	Explanation/reasoning
host.name	It is a host where the agent is running. Mainly used for cases when integration is installed on-prem. Note: we are not using host.id since it might be not unique
service.address	This field is present in case we provide a concrete target for scraping, like IP:PORT of some service (like mongodb), in some cases - it might be not needed to use this field
container.id	For now it is mainly used for: docker/containerd/kubernetes packages, this field is empty for other integrations. Container.id in this case is an id of the monitored container
cloud.account.id (new)	id used to identify different entities in a multi-tenant environment.
cloud.provider	To avoid minimal chance that account.id might be the same for different providers
cloud.region (new)	For services that are region specific
cloud.availability_zone (new)	For services that are zone specific
cloud.instance.id	host.name (can be defined manually by customer) is not unique enough. for Azure - instance.id is globally unique, AWS - region, GCP - availability zone. Technically for azure for example it would be enough to define cloud.instance.id only, but since it should be unified we include all fields: cloud.region/zone
agent.id	For cases when 2 data shipers are monitoring the same resource
cloud.project.id (deleted)	This value is empty with AWS and Azure cloud providers. cloud.account.id is hence considered

[rameshelastic]

@agithomas , @lalit-satapathy Has a separate issue been created for this around the merged schema? Is there value in keeping this issue open anymore?

[github-actions]

This PR is stale because it has been open for 60 days with no activity.