[AWS Farget] Set dimension fields

[constanca-m]

What does this PR do?

Set dimension fields for TSDB migration.

Details

ECS dimensions: The rationale for the defined ECS dimensions can be found here. The fields for this specific data stream would be:

• agent.id
• cloud.account.id
• cloud.region

For the specific fields, we would need these three:

• awsfargate.task_stats.cluster_name
• awsfargate.task_stats.identifier: each task is uniquely identified by the ID within a cluster
• Edit: container.name is added as dimension in case there is more than one container in a task.

Since ECS has a field that clusters four of them (account.id and region, cluster_name and identifier), container.labels.com_amazonaws_ecs_task-arn, we set that one as dimension instead of all four. The structure for that field is: arn:aws:ecs:<region>:<account-id>:task/<cluster-name>/<task-id>.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Related issues

Relates to #6732.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-07-07T07:33:32.783+0000

• Duration: 15 min 40 sec

Test stats 🧪

Test	Results
Failed	0
Passed	3
Skipped	0
Total	3

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (0/0)	💚
Files	100.0% (0/0)	💚
Classes	100.0% (0/0)	💚
Methods	66.667% (2/3)	👎 -33.333
Lines	100.0% (0/0)	💚 29.276
Conditionals	100.0% (0/0)	💚

[dmathieu]

This sets dimension fields, but also agent.id. Should we mention it here?

Suggested change

	- description: Set dimension fields.
	- description: Set `dimension`, and `agent.id` fields.

[constanca-m]

agent.id was already present, but this time, it is just explicitly set on the ecs.yml. I can add that. What do you think is best @dmathieu?

[dmathieu]

Mentioning that it's now explicitly set sounds good to me.

[constanca-m]

Thanks @dmathieu , I updated the changelog to include that

[tetianakravchenko]

• why cloud.account.id and region are not set as a dimensions?

  integrations/packages/awsfargate/data_stream/task_stats/sample_event.json

  Lines 136 to 138 in f2e2ac4

  	"cloud": {
  	"region": "us-west-2"
  	},

• can there be multiple clusters

  integrations/packages/awsfargate/data_stream/task_stats/sample_event.json

  Line 5 in f2e2ac4

  "cluster_name": "default",

  ? do you think it should be added as a dimension?

[tetianakravchenko]

wondering if awsfargate.task_stats.task_name isn't a better option for this

integrations/packages/awsfargate/data_stream/task_stats/sample_event.json

Line 133 in f2e2ac4

"task_name": "query-metadata"

so to rely not on the container name (in case there will be some changes in name generation), but the task name instead

[constanca-m]

I mention it on the description: The structure for that field is: arn:aws:ecs:<region>:<account-id>:task/<cluster-name>/<task-id>, that is why those fields are not set as dimension, because they are included in this field.

[elasticmachine]

Package awsfargate - 0.2.3 containing this change is available at https://epr.elastic.co/search?package=awsfargate