[Enhancement]: aws_db_instance - Validate whether IOPS meets gp3 baseline

[cdl-danielchapman]

Terraform Core Version

v1.3.5

AWS Provider Version

v4.46.0

Affected Resource(s)

No response

Expected Behavior

Storage increased beyond 400GB and IOPS/Throughput changed to match defaults of GP4

Actual Behavior

We have an issue where if we build an RDS instance at 100GiB specifying GP3 storage then the module dynamically sets IOPS to 3000 and throughput to 125MiB as expected

However, if we change the allocated_storage to 400GiB or more and apply that the apply fails with the below error

Error: updating RDS DB Instance (pg-prod-team-gp3): operation error RDS: ModifyDBInstance, https response error StatusCode: 400, RequestID: 970841cc-a412-4147-820c-700cb17e27fd, api error InvalidParameterCombination: Invalid iops value for engine name postgres and storage type gp3: 3000

If we build an RDS instance from a 100GiB snapshot but specify 400GiB in the terraform code at initial build it does the modification as expected and increases the storage/IOPS/throughput

But if we modify it after the instance has been built it errors. Ideally if it could dynamically lookup the values for IOPS/throughput that would be great. If they've been set in the code then set them to that but if they're null or not set to use the defaults i.e <400GiB 3K IOPS/125 MiB/S or >=400GiB 12K IOPS 500MiB/s

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

Error: updating RDS DB Instance (pg-prod-team-gp3): operation error RDS: ModifyDBInstance, https response error StatusCode: 400, RequestID: 970841cc-a412-4147-820c-700cb17e27fd, api error InvalidParameterCombination: Invalid iops value for engine name postgres and storage type gp3: 3000

  with module.rds_instance.aws_db_instance.rds,
  on .terraform/modules/rds_instance/main.tf line 1, in resource "aws_db_instance" "rds":
   1: resource "aws_db_instance" "rds" {

Steps to Reproduce

Build an RDS Postgres database with 100G allocated storage and GP3 as the storage type. Do not specify IOPS/throughput and this uses the defaults 3k/125Mb/s

Once built and available Increase the storage beyond 400G.

Debug Output

Error: updating RDS DB Instance (pg-prod-team-gp3): operation error RDS: ModifyDBInstance, https response error StatusCode: 400, RequestID: 970841cc-a412-4147-820c-700cb17e27fd, api error InvalidParameterCombination: Invalid iops value for engine name postgres and storage type gp3: 3000

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[justinretzolk]

Hey @cdl-danielchapman 👋 Thank you for taking the time to raise this! Given the numbers you're referencing, I'm sure you're familiar with the gp3 storage baselines, but I wanted to touch on that first, in case you or anyone in the future needs the info. The error message received is because once you cross over the 400GiB mark, the baseline for IOPS is 12000, rather than 3000, so setting an allocated_storage value of 400 and an iops value of 3000 is indeed invalid, as far as AWS is concerned.

That said, I think the more important bit here is why there's a difference in behavior when starting with 100 GiB of allocated storage and changing it to +400 vs starting with +400 GiB.

The key point here is that the iops argument is both Computed and Optional. This means that if it's not provided, it will be set based on the response from the action Terraform is taking (e.g. the response from AWS after creating the instance initially). What this means is that when a value is not set at creation time, whatever the IOPS defaults to at that time is what gets saved to the state for the iops argument's value.

Because of this Computed + Optional behavior, if you were to create an aws_db_instance instance with the storage_type set to gp3 at less than 400GiB of allocated storage, the iops argument would be saved to the state as 3000 (the baseline as defined in the AWS doc linked above). Similarly, at over 400GiB of allocated storage on initial creation, the iops would be set to 12000. This is why you had the difference in behavior depending on how much allocated storage you specified at time of creation.

On subsequent runs of terraform apply, when checking whether the iops argument has changes, we do currently verify that the allocated storage is above the minimum threshold for setting IOPS in gp3 before sending that in the update request. I'd like to leave this open as an enhancement request to explore the possibility of erroring if allocated storage is over 400GiB and IOPS are less than 12000.

[cdl-danielchapman]

@justinretzolk - your explanation makes perfect sense and thank you for responding.

Hopefully the enhancement can come to production, if there is anything we can do to help please reach out

[martinjzyang]

Is it possible to have an additional field called iops_offset and storage_throughput_offset rather than specifying the values? This way if storage is increased manually or through storage autoscaling, we won't error out. Furthermore tf plan will be explicit when changing storage above 400GiB will cause iops to change as well.

[mr-andres-carvajal]

@justinretzolk - Your answer is very helpful.

I have been looking through a lot of documentation and I cannot find the answer to the following:

We can also set max_allocated_storage - In the case where allocated_storage is < 400gb, but max allocated storage is > 400gb, and we set IOPs to 3k

What happens when our storage goes above 400gb? will the IOPs be scaled automatically to 12k? What will happen in subsequent TF plans?