[metricstransformprocessor] add new operation- convert_resource_attributes_to_metric_labels #1337

hossain-rayhan · 2020-10-21T05:50:31Z

Description:
This change will

add new operation convert_resource_attributes_to_metric_labels. By default, it converts all the resource attributes to metric labels. Customer can select specific resource attributes to set them as metric labels.
introduce batch operation for convert_resource_attributes_to_metric_labels. If customer does not set the value for metric_name (empty string), it means the changes will be applied to all metrics. However, currently, it only works and is tested for the convert_resource_attributes_to_labels operation. For other operations (e.g. update_label, add_label), it may not work properly. But, it shouldn't break existing user experience. Open issue to test and verify this change for other operations [metricstransformprocessor] Enable batch operation (all_metrics) for processor operations #1342

Link to tracking Issue:
#1297
#1343

Testing:

Added unit tests
Tested Manually

Documentation:
README updated

…butes_to_metric_labels Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>

anuraaga · 2020-10-21T06:37:36Z

processor/metricstransformprocessor/README.md


 ## Configuration
 ```yaml
 # transforms is a list of transformations with each element transforming a metric selected by metric name
 transforms:
  # name is used to match with the metric to operate on. This implementation doesn’t utilize the filtermetric’s MatchProperties struct because it doesn’t match well with what I need at this phase. All is needed for this processor at this stage is a single name string that can be used to match with selected metrics. The list of metric names and the match type in the filtermetric’s MatchProperties struct are unnecessary. Also, based on the issue about improving filtering configuration, it seems like this struct is subject to be slightly modified.
+
+  # new addition: now it accepts `all_metrics` as the value of 'metric_name' field. This batch operation applies changes to all the metrics. However, currently, it only works and is tested for the 'convert_resource_attributes_to_labels' operation. For other operations (e.g. update_label, add_label), it may not work properly. But, it shouldn't break existing user experience.


Instead of adding this in the README, capture it in the PR description and file an issue to fix for other operations in the future.

Created an issue: #1342

anuraaga · 2020-10-21T06:38:12Z

processor/metricstransformprocessor/README.md

@@ -55,6 +58,9 @@ transforms:
      aggregated_values: [values...]
      new_value: <new_value> 
      aggregation_type: {sum, mean, max}
+
+    # convert_resource_attributes_to_labels action converts all the resourde attributes to metric labels by default. Right now, it doesn't support selecting specific resource attributes as metric labels.


Ditto file an issue or update existing one to track the addition.

Created an issue: #1343

anuraaga · 2020-10-21T06:42:24Z

processor/metricstransformprocessor/metrics_transform_processor_testcases_test.go

@@ -570,6 +570,68 @@ var (
 					build(),
 			},
 		},
+		{


Is there a test case for the nil case?

Added a new test case for nil or empty resource attributes.

PettitWesley · 2020-10-21T19:26:00Z

processor/metricstransformprocessor/README.md


 ## Configuration
 ```yaml
 # transforms is a list of transformations with each element transforming a metric selected by metric name
 transforms:
  # name is used to match with the metric to operate on. This implementation doesn’t utilize the filtermetric’s MatchProperties struct because it doesn’t match well with what I need at this phase. All is needed for this processor at this stage is a single name string that can be used to match with selected metrics. The list of metric names and the match type in the filtermetric’s MatchProperties struct are unnecessary. Also, based on the issue about improving filtering configuration, it seems like this struct is subject to be slightly modified.
+
+  # it also accepts `all_metrics` as the value of 'metric_name' field. This batch operation applies changes to all the metrics. However, currently, it only works and is tested for the 'convert_resource_attributes_to_labels' operation.


nit: Can you just implement it for all operations in this PR? Or create a separate PR for just the all_metrics change

For most of the update operations, it should work fine. I already did some manual testing. Need more unit tests to verify it works fine for the operations we are claiming. Separate PR would make the process faster. Here is the issue I created for tracking- #1342 . Also, for the aggregation operations, I don't know how it will work. That's why planned for a separate PR.

anuraaga

Thanks, nice feature

anuraaga · 2020-10-21T23:35:28Z

@tigrannajaryan Can you review this PR? It seems nice to me to be able to have one processor to do resource / label handling since it's such a common use case but let us know your thoughts :)

Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>

hossain-rayhan · 2020-10-21T23:43:14Z

Sorry @anuraaga for pushing code after your approval. I just added the options for selecting specific resource attributes as metric labels. I feel it's a good option to have from the beginning and pretty simple to send with this PR. Would mind taking a look.

james-bebbington · 2020-10-21T23:44:55Z

processor/metricstransformprocessor/metrics_transform_processor.go

+			// for other operations. After updating the metrics, it will continue for the next iteration.
+			// TODO: add more unit tests and update README to pubicly anounce this feature for other
+			// operations as well.
+			if transform.MetricName == "all_metrics" {


Rather than use "all_metrics" as a special value, I think it would make more sense to do what @tigrannajaryan suggested and make MetricName optional in the config. If left out, then the transform applies to all metrics.

Pushed an update. Nice suggestion. Thanks @james-bebbington.

anuraaga

@james-bebbington's suggestion sounds nice, otherwise the new allow-list code LGTM

…ration for all metrics Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>

hossain-rayhan · 2020-10-22T16:48:04Z

@tigrannajaryan can we get this merged. Thanks.

tigrannajaryan

Requesting changes since I don't think this is the right approach. Please sync with @amanbrar1999 and @anuraaga

tigrannajaryan · 2020-10-22T16:44:34Z

processor/metricstransformprocessor/README.md


 ## Configuration
 ```yaml
 # transforms is a list of transformations with each element transforming a metric selected by metric name
 transforms:
  # name is used to match with the metric to operate on. This implementation doesn’t utilize the filtermetric’s MatchProperties struct because it doesn’t match well with what I need at this phase. All is needed for this processor at this stage is a single name string that can be used to match with selected metrics. The list of metric names and the match type in the filtermetric’s MatchProperties struct are unnecessary. Also, based on the issue about improving filtering configuration, it seems like this struct is subject to be slightly modified.
+
+  # if the value for `metric_name` is not set (empty string), the operations will be applied to all the metrics. However, currently, it only works and is tested for the 'convert_resource_attributes_to_labels' operation. Existing experience works fine with correct `metric_name`. 


it only works and is tested for the 'convert_resource_attributes_to_labels' operation.

This is not what I was expecting. I don't think we want a modification like this.

We discussed this in the SIG meeting yesterday where @amanbrar1999 and @anuraaga were present. My opinion is that it is the responsibility of exporters to do this transformation. If we want a processor to do this then I would expect it to be a separate specialized processor, provided that for whatever reason we are unable to fix the exporter. I don't see this fitting well in the metricstransformprocessor. The new functionality is not a metric transformation. It is a transformation of a resource into metric labels.

However, ultimately as I said I think this is a bug/deficiency of the Prometheus exporter.

it only works and is tested for the 'convert_resource_attributes_to_labels' operation.

**This is not what I was expecting. I don't think we want a modification like this.**
As I mentioned in the PR description, we created an issue [#1342] to work on that and it does not break existing user experience. I can work on that after this change or maybe with this change if community doesn't agree.

However, I don't clearly get why it should not go with metrictransform processor. In a sense, it's adding new labels to the metrics which means transforming the existing metrics.

Also, I will sync up with @anuraaga and @amanbrar1999.

To clarify myself- metricstransform processor already has an operation add_label and this incoming change is a similar operation which picks the labels from resource attributes. It's a simple new operation type which gives me a feeling to make it a part of this processor.

Also, our main target is awsemf exporter not the prometheus exporter. This tells us a strong point to keep this in the processor. What if our customer uses a new exporter where we don't have this feature. Then, we are kind of blocking our customers. Having this option in the processor will give our customer to choose any exporters they prefer.

Please correct me if I am wrong. Expecting a second thought @tigrannajaryan .

However, I don't clearly get why it should not go with metrictransform processor. In a sense, it's adding new labels to the metrics which means transforming the existing metrics.

This proposal adds something that does not fit nicely the metrictransform's intent of transforming metrics. This proposal converts resources into metric labels, which is not a metric transformation.

github-actions · 2020-11-03T05:17:26Z

This PR was marked stale due to lack of activity. It will be closed in 7 days.

hossain-rayhan · 2020-11-06T01:18:08Z

Closing this one as it we decided to implement a common utility for all exporters.
Issue: #1359
PR: 2060

…plicate code (#1337) Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>

* Rename sdk/trace/attributesMap.go -> sdk/trace/attributesmap.go Signed-off-by: Daniil Rutskiy <dstdfx@gmail.com> * Add missing tests for attributesMap Signed-off-by: Daniil Rutskiy <dstdfx@gmail.com> * Update CHANGELOG.md Signed-off-by: Daniil Rutskiy <dstdfx@gmail.com> * Add missing license header Signed-off-by: Daniil Rutskiy <dstdfx@gmail.com> * Delete underscores in test names Signed-off-by: Daniil Rutskiy <dstdfx@gmail.com> * Tests clean up Signed-off-by: Daniil Rutskiy <dstdfx@gmail.com>

[metricstransformprocessor] add new operation- convert_resource_attri…

e806ba0

…butes_to_metric_labels Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>

hossain-rayhan requested review from james-bebbington and a team as code owners October 21, 2020 05:50

github-actions bot assigned anuraaga Oct 21, 2020

project-bot bot added this to In progress in Collector Oct 21, 2020

anuraaga reviewed Oct 21, 2020

View reviewed changes

This was referenced Oct 21, 2020

[metricstransformprocessor] Enable batch operation (all_metrics) for processor operations #1342

Closed

[metricstransformprocessor] Select specific resource attributes to convert them to metric labels #1343

Closed

Update README and add a new unit test

bca2712

PettitWesley reviewed Oct 21, 2020

View reviewed changes

Collector automation moved this from In progress to Reviewer approved Oct 21, 2020

anuraaga approved these changes Oct 21, 2020

View reviewed changes

Select specific resource attributes to convert them to metric labels

7232818

Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>

james-bebbington reviewed Oct 21, 2020

View reviewed changes

anuraaga reviewed Oct 21, 2020

View reviewed changes

Make metric_name field optional and empty metric name means batch ope…

4dbdafa

…ration for all metrics Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>

tigrannajaryan requested changes Oct 22, 2020

View reviewed changes

Collector automation moved this from Reviewer approved to Review in progress Oct 22, 2020

hossain-rayhan mentioned this pull request Oct 23, 2020

[feature request] [exporter] resource_attributes_to_metric_labels - common utility for all Non-OTLP exporters #1359

Closed

github-actions bot added the Stale label Nov 3, 2020

hossain-rayhan closed this Nov 6, 2020

Collector automation moved this from Review in progress to Done Nov 6, 2020

dyladan referenced this pull request in dynatrace-oss-contrib/opentelemetry-collector-contrib Jan 29, 2021

ExporterHelper: Add ability to configure start function and remove du…

68c4e05

…plicate code (#1337) Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[metricstransformprocessor] add new operation- convert_resource_attributes_to_metric_labels #1337

[metricstransformprocessor] add new operation- convert_resource_attributes_to_metric_labels #1337

hossain-rayhan commented Oct 21, 2020 •

edited

Loading

anuraaga Oct 21, 2020

hossain-rayhan Oct 21, 2020

anuraaga Oct 21, 2020

hossain-rayhan Oct 21, 2020

anuraaga Oct 21, 2020

hossain-rayhan Oct 21, 2020

PettitWesley Oct 21, 2020

hossain-rayhan Oct 21, 2020

anuraaga left a comment

anuraaga commented Oct 21, 2020

hossain-rayhan commented Oct 21, 2020

james-bebbington Oct 21, 2020

hossain-rayhan Oct 22, 2020

anuraaga left a comment

hossain-rayhan commented Oct 22, 2020

tigrannajaryan left a comment

tigrannajaryan Oct 22, 2020

hossain-rayhan Oct 22, 2020

hossain-rayhan Oct 22, 2020

tigrannajaryan Oct 26, 2020

github-actions bot commented Nov 3, 2020

hossain-rayhan commented Nov 6, 2020

[metricstransformprocessor] add new operation- convert_resource_attributes_to_metric_labels #1337

[metricstransformprocessor] add new operation- convert_resource_attributes_to_metric_labels #1337

Conversation

hossain-rayhan commented Oct 21, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anuraaga left a comment

Choose a reason for hiding this comment

anuraaga commented Oct 21, 2020

hossain-rayhan commented Oct 21, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anuraaga left a comment

Choose a reason for hiding this comment

hossain-rayhan commented Oct 22, 2020

tigrannajaryan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Nov 3, 2020

hossain-rayhan commented Nov 6, 2020

hossain-rayhan commented Oct 21, 2020 •

edited

Loading