Unable to load up certain aws.service-quota Service Codes with Cloud Custodian

[CarloGiannattasio]

Describe the bug

In my org, I'm using Cloud Custodian to check the AWS service quotas. I have a script that goes through all the aws.service-quota servicecodes by updating a yml file then a runs the custodian run -s log -c <yml file> -r us-<region> -v for each of the service codes. Each loop of the script it runs one servicecode at a time.

The issue I'm facing that that, there are some service codes that still have this error:

Error:
botocore.errorfactory.InvalidParameterCombinationException: An error occurred (InvalidParameterCombination) when calling the GetMetricStatistics operation: You have requested up to 86400 datapoints, which exceeds the limit of 1440. You may reduce the datapoints requested by increasing Period, or decreasing the time range.

The service codes are:

• braket
• cognito-idp
• ecr
• elasticmapreduce
• kms
• monitoring
• rds
• rekognition
• servicequotas

Is there a way to have these service codes be able to be checked?

What did you expect to happen?

These and the rest of the service codes to run and email with services that hot the AWS limits

Cloud Provider

Amazon Web Services (AWS)

Cloud Custodian version and dependency information

Please copy/paste the following info along with any bug reports:

Custodian:   0.9.13
Python:      3.9.7 (v3.9.7:1016ef3790, Aug 30 2021, 16:39:15) 
             [Clang 6.0 (clang-600.0.57)]
Platform:    posix.uname_result(sysname='Darwin', nodename='C02DW2AVMD6M', release='19.6.0', version='Darwin Kernel Version 19.6.0: Thu Sep 16 20:58:47 PDT 2021; root:xnu-6153.141.40.1~1/RELEASE_X86_64', machine='x86_64')
Using venv:  True
Docker: False
Installed: 

argcomplete==1.12.3
attrs==21.2.0
boto3==1.17.102
botocore==1.20.102
importlib-metadata==4.6.0
jmespath==0.10.0
jsonschema==3.2.0
pyrsistent==0.18.0
python-dateutil==2.8.1
pyyaml==5.4.1
s3transfer==0.4.2
setuptools==57.0.0
six==1.16.0
tabulate==0.8.9
typing-extensions==3.10.0.0
urllib3==1.26.6
zipp==3.4.1
(nct-cloud-custodian) giannattasiog@C02DW2AVMD6M nct-cloud-custodian %

Policy

No response

Relevant log/traceback output

Error: 
botocore.errorfactory.InvalidParameterCombinationException: An error occurred (InvalidParameterCombination) when calling the GetMetricStatistics operation: You have requested up to 86400 datapoints, which exceeds the limit of 1440. You may reduce the datapoints requested by increasing Period, or decreasing the time range.

Extra information or context

Yml file

policies:
  - name: service-quota-usage-limit
    description: |
      find any services that have usage stats of over 80%
    resource: aws.service-quota
    filters:
      - type: value
        key: ServiceCode
        op: eq
        value: braket
      - UsageMetric: present
      - type: usage-metric
        limit: 5 # persentage, should be 80 according to the requirement
    actions:
      - type: notify
        to:
          - <my email>
        subject: "Service Quota Limit - [custodian <my AWS account> - <The AWS region>]"
        transport:
          type: sqs
          queue: https://sqs.us-east-1.amazonaws.com/<my AWS account>/CustodianNotifyQueue
          region: <The AWS region>

Python Script

#!/usr/bin/env python
import subprocess
import fileinput 
import os
import time

number = 0 
file = <yml file name>
path = <yml file path>
temp_word = 'value: TextChange'
temp_word_verify = 'value: TextChange'
last_word = 'TextChange'
serviceCode = [ last_word, "AWSCloudMap", "a4b", "access-analyzer", "account", "acm", "acm-pca", "airflow", "amplify", "apigateway", "appconfig", "appflow", "application-autoscaling", "application-cost-profiler", "appmesh", "apprunner", "appstream2", "appsync", "aps", "athena", "auditmanager", "autoscaling", "autoscaling-plans", "backup", "batch", "bugbust", "cassandra", "ce", "chatbot", "chime", "cloud9", "cloudformation", "cloud-front", "cloudhsm", "cloudsearch", "cloudshell", "cloudtrail", "codeartifact", "codebuild", "codecommit", "codedeploy", "codeguru-profiler", "codeguru-reviewer", "codepipeline", "cognito-identity", "cognito-sync", "comprehend", "comprehendmedical", "compute-optimizer", "connect", "crowdscale-usagelimitservice", "databrew", "dataexchange", "datapipeline", "datasync", "dax", "deeplens", "deepracer", "directconnect", "discovery", "dlm", "dms", "docdb", "ds", "dynamodb", "ebs", "ec2", "ecs", "eks", "elastic-inference", "elasticache", "elasticbeanstalk", "elasticfilesystem", "elasticloadbalancing", "elastictranscoder", "es", "events", "fargate", "finspace", "firehose", "fis", "fms", "forecast", "frauddetector", "fsx", "gamelift", "geo", "glacier", "globalaccelerator", "glue", "grafana", "greengrass", "guardduty", "iam", "imagebuilder","inspector", "inspector2", "iot", "iot1click", "iotanalytics", "iotcore", "iotdeviceadvisor", "iotevents", "iotfleethub", "iotsitewise", "iotthingsgraph", "iotwireless", "ivs", "kafka", "kendra", "kinesis", "kinesisanalytics","kinesisvideo", "lakeformation", "lambda", "launchwizard", "lex", "license-manager", "lightsail", "logs", "lookoutequipment", "lookoutmetrics", "lookoutvision",  "machinelearning", "macie", "macie2", "managedblockchain", "mediaconnect", "mediaconvert", "medialive", "mediapackage", "mediastore", "mediatailor", "mgn", "migrationhubstrategy",  "monitron", "mq", "neptune", "network-firewall", "networkinsights", "networkmanager", "nimble","opsworks", "opsworks-cm", "organizations", "panorama", "personalize", "pinpoint", "polly", "profile", "proton", "qldb", "quicksight", "ram", "redshift", "resource-groups", "robomaker", "route53", "route53resolver", "s3", "s3-outposts", "schemas", "secretsmanager", "securityhub", "serverlessrepo", "servicecatalog", "ses", "shield", "signer", "sms", "snow-device-management", "snowball", "sns", "sqs", "ssm", "ssm-contacts", "ssm-incidents", "sso", "states", "storagegateway", "sumerian", "support", "swf", "textract", "timestream","transcribe", "transfer", "translate", "vmimportexport", "vpc", "waf", "waf-regional", "wafv2", "wam", "workspaces", "xray", last_word]
serviceCodelen = len(serviceCode)


os.chdir(path)
open(file, "r+")
open_file = open(file, "r+")
filedata = fileinput.FileInput(file)


# Runs custodian for each service code. Each pass modifies the yml file
for code in serviceCode:
    with open(file, "r+") as f:
        content = f.read()
        f.seek(0)
        f.truncate()
        content = content.replace(str(temp_word), 'value: '+str(code))
        f.write(content)
        temp_word = 'value: '+code
        number += 1 
        print('--------------')
        print('--------------')
        print('Processing Service code '+ str(code) + ' : '+ str(number - 1) +' of '+str(serviceCodelen - 2))
        print('--------------')
        print('--------------')
        f.close()      
        subprocess.check_call(['custodian run -s log -c <yml file name> -r <The AWS region> -v '], shell=True)

[pstanton237]

I also experienced the same issue as you described.
The c7n command fails for the following services.

• "AWS Key Management Service (AWS KMS)"
• "Amazon CloudWatch Logs"
• "Amazon CloudWatch"
• "Amazon Cognito User Pools"
• "Amazon EMR"
• "Amazon Elastic Container Registry (Amazon ECR)"
• "Amazon Rekognition"
• "Amazon Relational Database Service (Amazon RDS)"
• "Service Quotas"

Below is my environment.

custodian version --debug

Please copy/paste the following info along with any bug reports:

Custodian:   0.9.13
Python:      3.9.7 (default, Sep 17 2021, 10:42:49) 
             [Clang 12.0.5 (clang-1205.0.22.9)]
Platform:    posix.uname_result(sysname='Darwin', nodename='jongyoungleeui-MacBookPro.local', release='21.1.0', version='Darwin Kernel Version 21.1.0: Wed Oct 13 17:33:23 PDT 2021; root:xnu-8019.41.5~1/RELEASE_X86_64', machine='x86_64')
Using venv:  True
Docker: False
Installed: 

argcomplete==1.12.3
attrs==21.2.0
boto3==1.17.102
botocore==1.20.102
importlib-metadata==4.6.0
jmespath==0.10.0
jsonschema==3.2.0
pyrsistent==0.18.0
python-dateutil==2.8.1
pyyaml==5.4.1
s3transfer==0.4.2
setuptools==58.2.0
six==1.16.0
tabulate==0.8.9
typing-extensions==3.10.0.0
urllib3==1.26.6
zipp==3.4.1

policies:
  - name: aws-service-quota-increase
    resource: aws.service-quota
    filters:
      - UsageMetric: present
      - type: usage-metric
        limit: 75

[pstanton237]

The c7n Quotas module seems to be called by specifying the query period as 24h when calling the get_metric_statistics function.
And some AWS services do not allow the CloudWatch metric data lookup for a 24h period.

So I changed the query period of the get_metric_statistics function to 15 minutes, and so far, I do not have a problem.
: pstanton237@e981f8e

Do you have any ideas?
Thanks,

[CarloGiannattasio]

I updated the version of Cloud Custodian and it seems to have the same metrics not working

[CarloGiannattasio]

Same error appears when setting the get_metric_statistics to 1 second

[rorynscott]

Is this bug associated with this PR? #7103

[CarloGiannattasio]

Sorry for the late response but yes

[kentnsw]

FYI, created a new PR #7140 to clear the CLA