[Azure - Billing Metrics] - Fails to ingest tags resource field

[muthu-mps]

Tags

In Azure tags is one of the essential field which allows us to organize the resources. The tags is used to group the billing data and categorize the cost by runtime environment.

Issue:

• I was trying to test the flattened type use case mentioned in this PR in billing metrics and found that the tags field is not available in Elasticsearch.
• The current Azure billing metrics module is not ingesting the tags field into Elasticsearch. The Azure billing API offers the tags attribute in the response. I am adding the Azure API link for reference.

Sample Billing Metrics [No tags]

Solution:

As the tags field is available in the API response value, we need to map the field from the response to the billing metrics module.

[botelastic]

This issue doesn't have a Team:<team> label.

[zmoog]

@muthu-mps are the tags available in the Usage Details API? If this is the case, are they available in both legacy and modern data formats?

[muthu-mps]

@zmoog - Yes, it's available as a part of the API response in both formats. Please find the sample response below,

[zmoog]

This is great. Supporting both legacy and modern formats sometimes requires jumping hoops, but this does look like a straightforward case.

[zmoog]

Do the Integrations team or you already have plans about taking on this?

[zmoog]

@muthu-mps I put together a simple PR to give getting tags from Usage Details API a try.

[zmoog]

I am able to get tags from others existing resources:

We already have an existing mapping available for azure.resource.tags.* from Azure module.

[lalit-satapathy]

I am able to get tags from others existing resources:

Since the changes is in beats now, assuming no additional code change needed for the tags to be visible in the integration run. Do we need to add a similar mapping in the integrations?

[muthu-mps]

I am able to get tags from others existing resources:

Since the changes is in beats now, assuming no additional code change needed for the tags to be visible in the integration run. Do we need to add a similar mapping in the integrations?

• We already have the mappings for tags defined in the package fields in the integrations. No need to make any changes. @zmoog please confirm.
• Once this change is available we can update the sample-events.json with tags in the integrations.
• @lalit-satapathy - Based on the above link referencing the beats mapping for tags field. I was able to find differences in the tags field for beats and integrations. Beats use the object type and the integrations are updated to the flattened type. The reason here is this is a meta field that is used across Azure modules. Are we considering updating the mapping in Beats to flattened type?
• Flattened type fields in billing integrations.

[zmoog]

• We already have the mappings for tags defined in the package fields in the integrations. No need to make any changes. @zmoog please confirm.

Yep, the mapping for the azure.resource.tags is available in most (and probably all) Azure Metrics integrations.

• Once this change is available we can update the sample-events.json with tags in the integrations.

Yep!

• @lalit-satapathy - Based on the above link referencing the beats mapping for tags field. I was able to find differences in the tags field for beats and integrations. Beats use the object type and the integrations are updated to the flattened type. The reason here is this is a meta field that is used across Azure modules. Are we considering updating the mapping in Beats to flattened type?
• Flattened type fields in billing integrations.

I see Azure integrations are not all using the same mapping for azure.resource.tags:

object:

• storage account
• monitor
• database account
• container service
• container registry
• container instance
• compute vm scaleset
• compute vm

flattened

• billing
• app insights
• app state

AWS is using the object type only.

I wonder why we picked different approaches.

What are the pros and cons of each mapping strategy for a key/value data type like tags? The object type probably gives users more flexibility and query options at the cost of a higher index size. On the contrary, flattened is storage efficient but may impose query limits.

Am I overlooking something important?