Adding a custom label to Temporal out-of-the-box metric

[goldtigerstone]

Is your feature request related to a problem? Please describe.
The default out-of-the-box metrics have been very helpful. Meanwhile, I thought adding custom labels (dimensions) to these metrics would offer more precise insights for troubleshooting.

For example, we are submitting thousands of WorkflowA instances, each with multiple subtype values. While temporal_workflow_failed is a useful metric for tracking failures, we would have better insights if we could add subtype as a custom label in the metric, such as temporal_workflow_failed{..., subtype=...}.

Describe the solution you'd like
When adding a custom Search Attribute, it would be helpful if Temporal provided an option to include these attributes as metric labels. This would make it easier to centralize and use custom Search Attributes directly in metric labels.

Describe alternatives you've considered
If it were possible to reference and copy the default out-of-the-box metrics within workflow or activity code, we could modify these metrics to include additional labels.

Additional context
Add any other context or screenshots about the feature request here.

[rodrigozhou]

Adding custom labels can be problematic. It could cause different namespaces to have a different set of labels, and potentially introduce very high cardinality, both of which can be difficult to scale.

[m4dvise]

@rodrigozhou Agreed, but will having a config globally specifying which search attributes to include and then setting the label to omited in namespaces where there is no data emitted for that SA solve the first issue? Also for second issue we can have a config for specifying values for the label to be emitted?