AWS::Logs::MetricFilter does not accept dimensions #64

srinivas-vangari · 2021-07-19T22:53:26Z

The following announcement makes dimensions available for metric filters.

https://aws.amazon.com/about-aws/whats-new/2021/05/amazon-cloudwatch-logs-announces-dimension-support-for-metric-filters/

However, it appears to be missing when specifying the resource through CloudFormation. Getting the following error on stack events console.

Encountered unsupported property Dimensions

The Python SDK documentation correctly shows this as available, but missing in CloudFormation docs for CloudWatchLogs.
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/logs.html#CloudWatchLogs.Client.put_metric_filter

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-logs-metricfilter.html

srinivas-vangari · 2021-07-21T18:43:18Z

This is a duplicate of aws-cloudformation/cloudformation-coverage-roadmap#419

arnulfojr · 2021-08-02T14:36:18Z

Any updates?

waifa · 2022-01-14T02:03:52Z

Working on this feature request. Support should be available shortly.

waifa · 2022-02-01T16:34:52Z

Deployment beginning shortly. I will update here once completed.

henhal · 2022-03-02T07:31:43Z

@waifa Any updates on this? 🙏

gshpychka · 2022-04-18T09:01:58Z

@waifa can you advise on the status of this?

waifa · 2022-04-18T19:28:51Z

The deployment from February failed unfortunately. We are currently working on a fix, thanks for your patience!

samjarman · 2022-05-10T01:04:39Z

@waifa Hey there! Curious if there are any updates on this? Thanks!

KirtiKunalShah · 2022-06-16T16:55:43Z

We have been working on issues related to deployment and traffic migration. Stay tuned to update which I should be able to provide by End of July

mbarneyjr · 2022-07-08T13:09:55Z

This seems to be live (at least defined) in the CloudFormation specification for us-east-1: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-resource-specification.html

jakob-keller · 2022-07-08T15:26:42Z

This seems to be live (at least defined) in the CloudFormation specification for us-east-1: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-resource-specification.html

I just checked in eu-central-1: It's not live as of now. However, it's possible to use the Unit property which is not yet covered in the CloudFormation documentation.

kgston · 2022-08-03T02:43:50Z

The documentation description for the Dimensions property is a little weird, but I've managed to successfully use it in ap-northeast-1

stevielb · 2022-08-12T17:51:45Z

@KirtiKunalShah You mentioned there might be an update around this time. Anything you can share with us on current progress?

jyotimahapatra · 2022-09-01T23:16:10Z

@waifa Can you help with a sample of how dimensions work? Looks like this is available in us-west-2 and i can consume it.
However, i get this error in Cfn Resource handler returned message: "Invalid request provided: AWS::Logs::MetricFilter" (RequestToken: abc41535-8432-1c18-e4ad-45438e510ff2, HandlerErrorCode: InvalidRequest)

Can you help with an example ?

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "CloudFormation template configuring AWS EKS Dataplane AZ metrics log groups",
    "Resources": {
        "KubeApiServerAzHealthzAPAusiw1az1MetricFilter1": {
            "Type": "AWS::Logs::MetricFilter",
            "Properties": {
                "LogGroupName": "abc",
                "FilterPattern": "{ $.xyz < 0.9 && $.AZ = usiw1-az1 }",
                "MetricTransformations": [{
                    "MetricValue": "1",
                    "MetricNamespace": "something",
                    "MetricName": "something",
                    "Dimensions": [{
                      "Key": "ab",
                      "Value": "de"
                    }]
                }],
            }
        }
    }
}

waifa · 2022-09-02T16:03:34Z

The dimension value needs to be a value matched in the filterPattern. Jyoti's problem was fixed by changing the value of the dimension in his template to "$.AZ".

waifa · 2022-09-02T16:21:32Z

Announcement: Dimensions and units are available in almost all regions. There are a select few regions that still need some work to make this feature available(like ap-northeast-3)!

I will close this issue once it is available in all regions.

jakob-keller · 2022-09-02T19:13:07Z

Announcement: Dimensions and units are available in almost all regions. There are a select few regions that still need some work to make this feature available(like ap-northeast-3)!

As of now, it is not available in eu-central-1:

CREATE_FAILED - Encountered unsupported property Dimensions

rene-kochen · 2022-09-07T10:44:01Z

It used to work in eu-central-1

Now our stacks no longer deploy:

12:40:53 | CREATE_FAILED | AWS::Logs::MetricFilter | MetricFilter
Encountered unsupported property Dimensions

waifa · 2022-09-08T20:42:06Z

Sincere apologies from AWS for this inconsistency. A mistake on our end caused the feature to be rolled back in eu-central-1 for a week and a couple other regions for a short period of time.

kathampy · 2022-10-12T09:20:22Z

The dimension value needs to be a value matched in the filterPattern. Jyoti's problem was fixed by changing the value of the dimension in his template to "$.AZ".

Can you support arbitrary values for dimension name? In my case the dimension name is not present in the log event and I need to specify it manually for the filter.

waifa · 2022-10-18T17:37:34Z

AWS::Logs::MetricFilter now accepts dimensions!

msambol · 2023-06-01T14:34:48Z

@waifa Do you know if it's on the roadmap to include a dimension value that is not in the filterPattern? For example, I want the Lambda function name to be a dimension but it's often not in the log and thus is not extracted by the filterPattern.

gordonpn · 2023-06-01T15:44:18Z

@msambol

@waifa Do you know if it's on the roadmap to include a dimension value that is not in the filterPattern? For example, I want the Lambda function name to be a dimension but it's often not in log and thus is not extracted by the filterPattern.

I raised this issue internally and trying to get it prioritized, but it's not currently on any roadmap.

r-heimann · 2023-06-06T10:56:34Z

Are you guys 100% sure that this is working in eu-central-1?

Resource handler returned message: "Invalid request provided: AWS::Logs::MetricFilter.
The specified filter pattern does not support dimensions
(Service: CloudWatchLogs, Status Code: 400, Request ID: a2271e5d-6464-4bfc-84db-XXXXXXXX)"
(RequestToken: 7b3cce6f-d6f0-8718-beb2-XXXXXXXX, HandlerErrorCode: InvalidRequest)

Example:

  ECSLogGroupFilter:
    Type: AWS::Logs::MetricFilter
    Properties: 
      FilterPattern: "?ERROR ?WARN"
      LogGroupName: !Ref ECSLogGroup
      MetricTransformations: 
        - MetricNamespace: "MetricNamespace"
          MetricName: "MetricName"
          Dimensions:
            - Key: "ClusterName"
              Value: !Ref ECSCluster
            - Key: "ServiceName"
              Value: !GetAtt ECSService.Name
          MetricValue: 1
          Unit: "Count"

This is working, by the way:

  ECSLogGroupFilter:
    Type: AWS::Logs::MetricFilter
    Properties: 
      FilterPattern: "?ERROR ?WARN"
      LogGroupName: !Ref ECSLogGroup
      MetricTransformations: 
        - MetricNamespace: "MetricNamespace"
          MetricName: "MetricName"
          MetricValue: 1
          Unit: "Count"

gordonpn · 2023-06-06T14:38:30Z

@r-heimann

Are you guys 100% sure that this is working in eu-central-1?

Yes, it should be working in eu-central-1.

I believe your Metric Filter is returning an error because you are using a text-based filter pattern (as opposed to space-delimited or JSON). Text-based filter patterns do not support dimensions.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/FilterAndPatternSyntax.html#logs-metric-filters-dimensions:~:text=publish%20dimensions%20with%20metrics%20that%20JSON%20and%20space%2Ddelimited%20log%20events%20generate

r-heimann · 2023-06-07T05:53:43Z

Yes, it should be working in eu-central-1.

I believe your Metric Filter is returning an error because you are using a text-based filter pattern (as opposed to space-delimited or JSON). Text-based filter patterns do not support dimensions.

I see, i have no idea why such a limitation exists in the first place but i'll try to get it working using your link.

Ricketiman · 2023-09-10T02:23:07Z

I raised this issue internally and trying to get it prioritized, but it's not currently on any roadmap.

@gordonpn it would be great to include this feature on roadmap for the use cases it can support :)

alancnet · 2023-09-11T20:19:47Z

I should be able to hard code a dimension, rather than always pull it from a log line.

Use cases:

Filtering lambda logs that don't contain the function name.
Attributing logs to deployment environments (prod, qa, etc)

This could be easily achieved by allowing non-selectors as dimension values (strings that don't start with $)

gordonpn · 2023-09-11T20:46:46Z

I should be able to hard code a dimension, rather than always pull it from a log line.

Use cases:

Filtering lambda logs that don't contain the function name.

Attributing logs to deployment environments (prod, qa, etc)

This could be easily achieved by allowing non-selectors as dimension values (strings that don't start with $)

@alancnet I agree with you and this is also something I raised internally, which is also pending prioritization.

brentcetinich · 2023-10-10T07:36:26Z

I should be able to hard code a dimension, rather than always pull it from a log line.

Use cases:

Filtering lambda logs that don't contain the function name.

Attributing logs to deployment environments (prod, qa, etc)

This could be easily achieved by allowing non-selectors as dimension values (strings that don't start with $)

Raised this issue since this issue is not solved still.

waifa mentioned this issue Jan 14, 2022

Adding Logs MetricFilter Dimensions and Unit Support #84

Merged

waifa self-assigned this Jan 18, 2022

simonwiesmann8732 mentioned this issue Feb 2, 2022

AWS::Logs::MetricFilter - Dimension support aws-cloudformation/cloudformation-coverage-roadmap#419

Closed

peterwoodworth mentioned this issue Apr 11, 2022

aws-cdk/aws-logs: Cannot add dimensions to MetricFilter aws/aws-cdk#16999

Closed

2 tasks

HWong-sudo mentioned this issue Jul 30, 2022

Add custom metric filters for common scaling errors to default cloudwatch dashboard aws/aws-parallelcluster#4195

Merged

4 tasks

waifa closed this as completed Oct 18, 2022

AWS::Logs::MetricFilter does not accept dimensions #64

AWS::Logs::MetricFilter does not accept dimensions #64

srinivas-vangari commented Jul 19, 2021

srinivas-vangari commented Jul 21, 2021

arnulfojr commented Aug 2, 2021

waifa commented Jan 14, 2022

waifa commented Feb 1, 2022

henhal commented Mar 2, 2022

gshpychka commented Apr 18, 2022

waifa commented Apr 18, 2022

samjarman commented May 10, 2022

KirtiKunalShah commented Jun 16, 2022

mbarneyjr commented Jul 8, 2022 •

edited

jakob-keller commented Jul 8, 2022

kgston commented Aug 3, 2022 •

edited

stevielb commented Aug 12, 2022

jyotimahapatra commented Sep 1, 2022

waifa commented Sep 2, 2022

waifa commented Sep 2, 2022 •

edited

jakob-keller commented Sep 2, 2022

rene-kochen commented Sep 7, 2022

waifa commented Sep 8, 2022

kathampy commented Oct 12, 2022

waifa commented Oct 18, 2022

msambol commented Jun 1, 2023 •

edited

gordonpn commented Jun 1, 2023 •

edited

r-heimann commented Jun 6, 2023 •

edited

gordonpn commented Jun 6, 2023

r-heimann commented Jun 7, 2023

Ricketiman commented Sep 10, 2023

alancnet commented Sep 11, 2023

gordonpn commented Sep 11, 2023

brentcetinich commented Oct 10, 2023

AWS::Logs::MetricFilter does not accept dimensions #64

AWS::Logs::MetricFilter does not accept dimensions #64

Comments

srinivas-vangari commented Jul 19, 2021

srinivas-vangari commented Jul 21, 2021

arnulfojr commented Aug 2, 2021

waifa commented Jan 14, 2022

waifa commented Feb 1, 2022

henhal commented Mar 2, 2022

gshpychka commented Apr 18, 2022

waifa commented Apr 18, 2022

samjarman commented May 10, 2022

KirtiKunalShah commented Jun 16, 2022

mbarneyjr commented Jul 8, 2022 • edited

jakob-keller commented Jul 8, 2022

kgston commented Aug 3, 2022 • edited

stevielb commented Aug 12, 2022

jyotimahapatra commented Sep 1, 2022

waifa commented Sep 2, 2022

waifa commented Sep 2, 2022 • edited

jakob-keller commented Sep 2, 2022

rene-kochen commented Sep 7, 2022

waifa commented Sep 8, 2022

kathampy commented Oct 12, 2022

waifa commented Oct 18, 2022

msambol commented Jun 1, 2023 • edited

gordonpn commented Jun 1, 2023 • edited

r-heimann commented Jun 6, 2023 • edited

gordonpn commented Jun 6, 2023

r-heimann commented Jun 7, 2023

Ricketiman commented Sep 10, 2023

alancnet commented Sep 11, 2023

gordonpn commented Sep 11, 2023

brentcetinich commented Oct 10, 2023

mbarneyjr commented Jul 8, 2022 •

edited

kgston commented Aug 3, 2022 •

edited

waifa commented Sep 2, 2022 •

edited

msambol commented Jun 1, 2023 •

edited

gordonpn commented Jun 1, 2023 •

edited

r-heimann commented Jun 6, 2023 •

edited