resourceAwsSsmParameterRead is slow even with increased SSM parameter rate-limit

[rafalcieslak]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Using a lot of aws_ssm_parameters takes a very long time to refresh, even with increased SSM rate-limit requested from AWS.

In our specific scenario where we use a few hundereds of parameters, the refresh stage of terraform apply takes ~90 seconds w/o parameters, or ~6 minutes with parameters.

AWS provides a mechanism to increase the rate limit for SSM parameter requests, but enabling this feature does not resolve this issue. The reason for that is that this increased rate limit does not apply to all SSM API actions. In particular, it applies to GetParameter, but not to ListTagsForResource.

This issue is very easy to reproduce - just create 100 aws_ssm_parameters, apply, then then measure terraform refresh time. Such simple configuration usually takes ~60 seconds to refresh.

The resourceAwsSsmParameterRead function uses GetParameter, DescribeParameters and ListTagsForResource. In order to leverage the increased rate limit, the read function would need to not use the last (or two last) of these.

Obviously it would unreasonable to request that this resource does not check for parameter's tags with AWS, as this is obviously an important feature.

Possible ways to address this problem include:

• Adding a configuration flag to the resource (e.g. ignore_tags), which, if set to true, prevents tags from being read during refresh. Note that this is different than terraform's ignore_changes lifecycle option, as that doesn't affect what the read function does, it only affects how terraform compares resources.
• Adding a configuration flag to the resource (e.g. fast_mode), which, if set to true, reduces the read function to just GetParameter. In this mode only the value of the parameter is checked with AWS - which will be tremendously fast even for a large number of parameters, but has the obvious disadvantage of not detecting drifts in tags or parameter tiers.
• Implementing a separate resource (e.g. aws_ssm_parameter_fast) which is a simpler version of aws_ssm_parameter which does not support tags nor tiers. I think this approach does not lie well with the style of AWS provider and might introduce unnecessary complexity and confusion.
• This could also be partially resolved in terraform by implementing a lifecycle flag that disables refresh actions for a resource. This would allow users to mark some resources as "never need a refresh", and thus the time-consuming parameter refreshing would be avoided.

I might be able to contribute an implementation of a resolution of this problem, but it certainly requires a solid discussion first.

Affected Resource(s)

• aws_ssm_parameter

[hlarsen]

Is this still an issue? I'm starting to debug some slow plans we seem to be facing - I haven't dug that far into it yet, but moving some sensitive values to Param Store is one of the big changes we've made recently.

[a1031]

I've just recently added a single aws_ssm_parameter resource to my config and my plans are now taking anywhere from 30-60 seconds. I'd noticed that the plan is consistently stalling the Refreshing state... operation only for this aws_ssm_parameter resource, which led me to this issue.

For now I think I will just switch from SSM Parameter Store to Secrets Manager. Although the data I'm storing is not sensitive, both services operate (and probably cost) around the same, but I've never had these issues with SM.