Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ASGHandler - ClientError (ValidationError) when using a high number of configured schedules #549

Closed
6 tasks
FugroEgger opened this issue Jun 11, 2024 · 2 comments
Labels

Comments

@FugroEgger
Copy link

FugroEgger commented Jun 11, 2024

Describe the bug
Stack region: eu-west-1
Stack-Name: cs-instance-scheduler
TagName: scheduler_period
UsingAWSOrganizations: Yes
regions: ap-southeast-1,ap-southeast-2,eu-central-1,eu-west-1,eu-west-3,eu-north-1,me-south-1,me-central-1,us-east-1,us-east-2,us-west-1
63 possible schedules

This ASGHandler (action: scheduler:run) in Hub account shows below error for multiple accounts/regions where ASG is not used (no groups configured).
ASG scheduling works if ASG is tagged

Error message

[ERROR] ClientError: An error occurred (ValidationError) when calling the DescribeAutoScalingGroups operation: The request isn't valid. The number of values specified for the tag:scheduler_period filter type exceeds the maximum of 5 tag:scheduler_period.
Traceback (most recent call last):
  File "/var/task/aws_lambda_powertools/logging/logger.py", line 450, in decorate
    return lambda_handler(event, context, *args, **kwargs)
  File "/var/task/instance_scheduler/handler/asg.py", line 68, in lambda_handler
    [num_tagged_auto_scaling_groups, num_schedules] = schedule_auto_scaling_groups(
  File "/var/task/instance_scheduler/handler/asg.py", line 135, in schedule_auto_scaling_groups
    for group in asg_service.get_schedulable_groups(schedule_names):
  File "/var/task/instance_scheduler/service/asg.py", line 263, in get_schedulable_groups
    for page in paginator.paginate(Filters=filters):
  File "/var/lang/lib/python3.11/site-packages/botocore/paginate.py", line 269, in __iter__
    response = self._make_request(current_kwargs)
  File "/var/lang/lib/python3.11/site-packages/botocore/paginate.py", line 357, in _make_request
    return self._method(**current_kwargs)
  File "/var/lang/lib/python3.11/site-packages/botocore/client.py", line 553, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/lang/lib/python3.11/site-packages/botocore/client.py", line 1009, in _make_api_call
    raise error_class(parsed_response, operation_name)

To Reproduce

  • create 10 schedules with periods
  • hub account uses (UsingAWSOrganizations)
  • hub account uses at least 2 regions
  • deploy remote stack in spoke account
  • create an autoscaling group in region1 (no scheduling tag needed)
  • after execution check cloudwatch ASGHandler log group manually or run this log insight query:
fields @timestamp, @message, @logStream, @log
| filter (@message like "ClientError:")
| sort @timestamp desc
| limit 1000

Expected behavior
Number of values specified for the tag:scheduler_period should not be limited.

Please complete the following information about the solution:

  • Version: 3.0.0

  • Region: eu-west-1 (for hub and spoke stack deployment)

  • Was the solution modified from the version published on this repository? No

  • If the answer to the previous question was yes, are the changes available on GitHub?

  • Have you checked your service quotas for
    the sevices this solution uses? No

  • Were there any errors in the CloudWatch Logs? see above and attachment
    Troubleshooting

Screenshots
none

Additional context
All accounts are managed by Controltower,
Minor SCP are active

ASGHandler_call.txt
Cloudwatch_error.txt
Cloudformation was used to create all schedules
Iac_schedule_definition.template.txt

@FugroEgger FugroEgger added the bug label Jun 11, 2024
@FugroEgger FugroEgger changed the title ClientError (ValidationError) ASG scheduling with more than 5 schedules in ASGHandler ASGHandler - ClientError (ValidationError) when using a high number of configured schedules Jun 11, 2024
@CrypticCabub
Copy link
Member

Thanks for reporting this!

This issue should only occur when creating more than 5 schedules within a short amount of time, I have added it to our backlog to be fixed in an upcoming patch release.

@CrypticCabub
Copy link
Member

Fixed in v3.0.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants