[Bug]: After switching an RDS instance from gp2 to gp3, following apply will fail if IOPS is not specified

[etiennechabert]

Terraform Core Version

1.2.3

AWS Provider Version

4.46.0

Affected Resource(s)

• aws_db_instance

Expected Behavior

If specifying iops is mandatory when choosing gp3:

• You should not be able to apply the change without specifying it, a validator is missing

If IOPS is NOT mandatory when choosing gp3:

• You should not be blocked during further apply, and your plan should not describe a change about iops

Actual Behavior

The first apply was successful, allowing me to switch an existing RDS instance from GP2 to GP3. But now my pipeline is blocked with the following plan/error

  ~ resource "aws_db_instance" "this" {
        id                                    = "a_db"
      ~ iops                                  = 3000 -> 0
        name                                  = "a_db"

....

│ Error: updating RDS DB Instance (a_db): operation error RDS: ModifyDBInstance, https response error StatusCode: 400, RequestID: 61792e08-c93b-4834-a35c-eb029b29ba30, api error InvalidParameterCombination: You can't specify IOPS or storage throughput for engine postgres and a storage size less than 400.

I can work around the error by adding to my resource the optional iops parameter with a value of 3000, unblocking my pipeline by avoiding the change:

  resource "aws_db_instance" "this" {
      id                                    = "a_db"
      iops                                  = 3000
      name                                  = "a_db"
        
...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

resource "aws_db_instance" "default" {
  ...
  allocated_storage  = 20
  storage_type       = "gp2"
}

After first apply (you have an RDS instance on gp2), change storage_type, and apply two times: after the 1st apply you are on gp3, but now the 2nd (and following) applies will fail:

resource "aws_db_instance" "default" {
  ...
  allocated_storage  = 20
  storage_type       = "gp3"
}

Steps to Reproduce

• Create an RDS instance (postgres might be important), with a gp2 volume of 20 GB
• After first apply, switch it to gp3 without specifying the IOPS parameter
• Try to apply again, you should now see an unexpected change:
  • 0 → 3000 IOPS
  • Your apply will fail with the following error:
  • api error InvalidParameterCombination: You can't specify IOPS or storage throughput for engine postgres and a storage size less than 400

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[etiennechabert]

Related to:

• [Enhancement]: Add gp3 storage_type support to aws_db_instance #27702
• [Bug]: With gp3 storage_type for aws_db_instance cannot increase storage with gp3, or change to gp3, with storage < 400GB #28160

Should have been fixed by:

• r/aws_db_instance: Fix error modifying allocated_storage when storage_type is "gp3" #28243

[ewbankkit]

@etiennechabert Thanks for raising this issue.
In your Terraform configuration do you have an explicit iops = 0 or are you omitting that attribute completely?

[etiennechabert]

@ewbankkit during the first apply, the one that succeeded initially, switching my instance from gp2 to gp3, my terraform configuration was not specifying anything for IOPS... but I realize now that the module we are using is setting iops to a default value of 0:

• https://registry.terraform.io/modules/terraform-aws-modules/rds/aws/latest/submodules/db_instance#input_iops
• Effectively, I am having iops = 0 sent to the aws_db_instance resource because of this ⚠️

First apply

• In my plan during this apply, there were no changes regarding iops described
  • That's probably because at this point, my database is still a gp2, therefor the remote IOPS is equal to 0
• The apply is successful, my database is now using a gp3 volume

Following applies

During the following apply, the ones that failed, my terraform configuration was still not defining iops, but the default value of terraform-aws-modules/rds been 0, this is now probably clashing with the remote state of my database and causing the change:

• My plan is now showing the following change iops = 3000 -> 0
• The apply is failing with the error: You can't specify IOPS or storage throughput for engine postgres and a storage size less than 400.
  • Kind of unrelated error with the actual issue in my opinion.
• My pipeline is blocked

Workaround

I am now forced to set iops = 3000 in my Terraform configuration to avoid the invalid change, unblocking my pipeline.

Proposal to avoid this

Should the provider do a max(3000, var.iops) when allocated_storage == 'gp3'?

My understanding is that a gp3 volume cannot have less than 3000 IOPS according to the documentation, that would probably avoid this issue all together.

Open question

With that been said, I am not sure anymore if this problem should actually be reported to https://github.com/terraform-aws-modules/terraform-aws-rds or is actually about the AWS Provider.

[etiennechabert]

Actually reading one more time the documentation of the module we are using, and more specifically about storage_type, I think I can answer this question myself: https://github.com/terraform-aws-modules/terraform-aws-rds/blob/master/modules/db_instance/variables.tf#L24-L28

If you specify 'io1' or 'gp3' , you must also include a value for the 'iops' parameter...

Seems like by switching from gp2 to gp3, I was now using the module the wrong way by not specifying iops... and that my workaround is actually what is needed as documented.

Happy to close this issue if you say so.

[ewbankkit]

@etiennechabert Thanks for the response.
Let's leave the issue open a while and see if anyone else from the community has input.

[TomWizen]

We are facing the same issue. When trying to configure a migration from gp2 to gp3 with iops parameter it is failing with the above error. The only possible way to do it right now it to separate it to two terraform runs, which is not ideal.

[etiennechabert]

@TomWizen can you please share if you are using this module on top of the AWS Provider: https://github.com/terraform-aws-modules/terraform-aws-rds

[etiennechabert]

I started over again all my tests, this time directly using aws_db_instance.

In this messages my results for creation tests, in next message I will put my transition test results:

• Create a mariadb with gp3 🟢
• Create a mariadb with gp3 while setting IOPS to 3000 🟥
• Create a mariadb with gp3 while setting IOPS to 3000 and storage_throughput to 125 🟥

Note: same results with a postgres

Create a mariadb with gp3: 🟢

resource "aws_db_instance" "default_gp3" {
  identifier           = "default-gp3"
  allocated_storage    = 20
  db_name              = "direct_gp3"
  engine               = "mariadb"
  engine_version       = "10.3"
  instance_class       = "db.t3.micro"
  username             = "foo"
  password             = "foobarbaz"
  parameter_group_name = "default.mariadb10.3"
  skip_final_snapshot  = true
  apply_immediately    = true
  storage_type         = "gp3"
}

Just works

Create a mariadb with gp3 while setting IOPS to 3000 🟥

resource "aws_db_instance" "default_gp3" {
  identifier           = "default-gp3"
  allocated_storage    = 20
  db_name              = "direct_gp3"
  engine               = "mariadb"
  engine_version       = "10.3"
  instance_class       = "db.t3.micro"
  username             = "foo"
  password             = "foobarbaz"
  parameter_group_name = "default.mariadb10.3"
  skip_final_snapshot  = true
  apply_immediately    = true
  storage_type         = "gp3"
  iops                 = 3000
}

Error:

InvalidParameterCombination: You can't specify IOPS or storage throughput for engine mariadb and a storage size less than 400

Create a mariadb with gp3 while setting IOPS to 3000 and storage_throughput to 125 🟥

resource "aws_db_instance" "default_gp3" {
  identifier           = "default-gp3"
  allocated_storage    = 20
  db_name              = "direct_gp3"
  engine               = "mariadb"
  engine_version       = "10.3"
  instance_class       = "db.t3.micro"
  username             = "foo"
  password             = "foobarbaz"
  parameter_group_name = "default.mariadb10.3"
  skip_final_snapshot  = true
  apply_immediately    = true
  storage_type         = "gp3"
  iops                 = 3000
  storage_throughput   = 125
}

Error:

InvalidParameterCombination: You can't specify IOPS or storage throughput for engine mariadb and a storage size less than 400

I find this test interesting, because in the end, this is exactly the state of the database succesfully created in my first test (default_gp3), but you cannot create a DB with exactly these parameters 🤔

Note

I know that the creation is not what was initially the topic of this issue, but I still find it interesting and probably connected to this issue

[etiennechabert]

Now about transitions:

• Create a mariadb with gp2 and transition it to gp3 🟢
• Create a mariadb with gp2 and transition it to gp3 and then change instance_class 🟢
• Create a mariadb with gp2 and transition it to gp3 with IOPS to 3000 🟥

Create a mariadb with gp2 and transition it to gp3 🟢

Start with

resource "aws_db_instance" "default_gp2_to_gp3" {
  identifier           = "default-gp2-to-gp3"
  allocated_storage    = 20
  db_name              = "gp2_to_gp3"
  engine               = "mariadb"
  engine_version       = "10.3"
  instance_class       = "db.t3.micro"
  username             = "foo"
  password             = "foobarbaz"
  parameter_group_name = "default.mariadb10.3"
  skip_final_snapshot  = true
  apply_immediately    = true
  storage_type         = "gp2"
}

Then change

  storage_type         = "gp3"

Works

Create a mariadb with gp2 and transition it to gp3 and then change instance_class 🟢

Start with

resource "aws_db_instance" "default_gp2_to_gp3" {
  identifier           = "default-gp2-to-gp3"
  allocated_storage    = 20
  db_name              = "gp2_to_gp3"
  engine               = "mariadb"
  engine_version       = "10.3"
  instance_class       = "db.t3.micro"
  username             = "foo"
  password             = "foobarbaz"
  parameter_group_name = "default.mariadb10.3"
  skip_final_snapshot  = true
  apply_immediately    = true
  storage_type         = "gp2"
}

Then change

  storage_type         = "gp3"

Then change

  instance_class         = "db.t3.small"

Works, with this test I wanted to make sure that it was still possible to modify this DB, while not providing iops/storage_throughput

Create a MariaDB with gp2 and transition it to gp3 with IOPS to 3000 🟥

Start with

resource "aws_db_instance" "default_gp2_to_gp3" {
  identifier           = "default-gp2-to-gp3"
  allocated_storage    = 20
  db_name              = "gp2_to_gp3"
  engine               = "mariadb"
  engine_version       = "10.3"
  instance_class       = "db.t3.micro"
  username             = "foo"
  password             = "foobarbaz"
  parameter_group_name = "default.mariadb10.3"
  skip_final_snapshot  = true
  apply_immediately    = true
  storage_type         = "gp2"
}

Then change

  storage_type         = "gp3"
  iops                 = 3000

Error:

InvalidParameterCombination: You can't specify IOPS or storage throughput for engine mariadb and a storage size less than 400

Note: same result with

  storage_type         = "gp3"
  iops                 = 3000
  storage_throughput   = 125