Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

resourceAwsSsmParameterRead is slow even with increased SSM parameter rate-limit #9925

Open
rafalcieslak opened this issue Aug 30, 2019 · 3 comments
Labels
enhancement Requests to existing resources that expand the functionality or scope. service/ssm Issues and PRs that pertain to the ssm service. stale Old or inactive issues managed by automation, if no further action taken these will get closed.

Comments

@rafalcieslak
Copy link

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Using a lot of aws_ssm_parameters takes a very long time to refresh, even with increased SSM rate-limit requested from AWS.

In our specific scenario where we use a few hundereds of parameters, the refresh stage of terraform apply takes ~90 seconds w/o parameters, or ~6 minutes with parameters.

AWS provides a mechanism to increase the rate limit for SSM parameter requests, but enabling this feature does not resolve this issue. The reason for that is that this increased rate limit does not apply to all SSM API actions. In particular, it applies to GetParameter, but not to ListTagsForResource.

This issue is very easy to reproduce - just create 100 aws_ssm_parameters, apply, then then measure terraform refresh time. Such simple configuration usually takes ~60 seconds to refresh.

The resourceAwsSsmParameterRead function uses GetParameter, DescribeParameters and ListTagsForResource. In order to leverage the increased rate limit, the read function would need to not use the last (or two last) of these.

Obviously it would unreasonable to request that this resource does not check for parameter's tags with AWS, as this is obviously an important feature.

Possible ways to address this problem include:

  • Adding a configuration flag to the resource (e.g. ignore_tags), which, if set to true, prevents tags from being read during refresh. Note that this is different than terraform's ignore_changes lifecycle option, as that doesn't affect what the read function does, it only affects how terraform compares resources.
  • Adding a configuration flag to the resource (e.g. fast_mode), which, if set to true, reduces the read function to just GetParameter. In this mode only the value of the parameter is checked with AWS - which will be tremendously fast even for a large number of parameters, but has the obvious disadvantage of not detecting drifts in tags or parameter tiers.
  • Implementing a separate resource (e.g. aws_ssm_parameter_fast) which is a simpler version of aws_ssm_parameter which does not support tags nor tiers. I think this approach does not lie well with the style of AWS provider and might introduce unnecessary complexity and confusion.
  • This could also be partially resolved in terraform by implementing a lifecycle flag that disables refresh actions for a resource. This would allow users to mark some resources as "never need a refresh", and thus the time-consuming parameter refreshing would be avoided.

I might be able to contribute an implementation of a resolution of this problem, but it certainly requires a solid discussion first.

Affected Resource(s)

  • aws_ssm_parameter
@rafalcieslak rafalcieslak added the enhancement Requests to existing resources that expand the functionality or scope. label Aug 30, 2019
@ghost ghost added the service/ssm Issues and PRs that pertain to the ssm service. label Aug 30, 2019
@github-actions github-actions bot added the needs-triage Waiting for first response or review from a maintainer. label Aug 30, 2019
@hlarsen
Copy link

hlarsen commented Aug 17, 2021

Is this still an issue? I'm starting to debug some slow plans we seem to be facing - I haven't dug that far into it yet, but moving some sensitive values to Param Store is one of the big changes we've made recently.

@breathingdust breathingdust removed the needs-triage Waiting for first response or review from a maintainer. label Sep 22, 2021
@a1031
Copy link

a1031 commented Feb 20, 2022

I've just recently added a single aws_ssm_parameter resource to my config and my plans are now taking anywhere from 30-60 seconds. I'd noticed that the plan is consistently stalling the Refreshing state... operation only for this aws_ssm_parameter resource, which led me to this issue.

For now I think I will just switch from SSM Parameter Store to Secrets Manager. Although the data I'm storing is not sensitive, both services operate (and probably cost) around the same, but I've never had these issues with SM.

Copy link

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

@github-actions github-actions bot added the stale Old or inactive issues managed by automation, if no further action taken these will get closed. label Dec 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Requests to existing resources that expand the functionality or scope. service/ssm Issues and PRs that pertain to the ssm service. stale Old or inactive issues managed by automation, if no further action taken these will get closed.
Projects
None yet
Development

No branches or pull requests

4 participants