[Bug]: aws_db_instance ca_cert_identifier field applied but not reflected

[valugi]

Terraform Core Version

1.5.7

AWS Provider Version

5.14.0

Affected Resource(s)

aws_db_instance

Expected Behavior

RDS AWS instance should have rds-ca-rsa2048-g1 CA.

Actual Behavior

RDS AWS instance remains on rds-ca-2019 .

Relevant Error/Panic Output Snippet

There is no error. The changed is detected correctly in plan and apply. According to terraform the change is done, but the value is not changed. 

Rebooting the db does not help. No items are left to install in maintenance.

Terraform Configuration Files

resource "aws_db_instance" "db" {
  identifier_prefix                   = "${lower("${var.name}-RDS")}-"
  allocated_storage                   = var.allocated_storage
  max_allocated_storage               = var.max_allocated_storage
  storage_type                        = var.storage_type
  engine                              = var.db_engine
  engine_version                      = var.db_engine_version
  storage_encrypted                   = var.storage_encrypted
  instance_class                      = var.instance_class
  username                            = var.username
  manage_master_user_password         = true
  iam_database_authentication_enabled = true
  db_subnet_group_name                = aws_db_subnet_group.default.name
  multi_az                            = var.multi_az
  copy_tags_to_snapshot               = true
  final_snapshot_identifier           = "${lower("${var.name}-RDS")}-final-snapshot"
  vpc_security_group_ids              = [aws_security_group.rds_sg.id]
  parameter_group_name                = var.parameter_group_name
  backup_retention_period             = "7"
  backup_window                       = "05:00-06:00"
  maintenance_window                  = "Sun:06:00-Sun:07:00"
  auto_minor_version_upgrade          = "true"
  allow_major_version_upgrade         = var.allow_major_version_upgrade
  monitoring_interval                 = var.monitoring_interval
  monitoring_role_arn                 = var.monitoring_role_arn
  deletion_protection                 = var.deletion_protection
  enabled_cloudwatch_logs_exports     = var.enabled_cloudwatch_logs_exports
  ca_cert_identifier                  = "rds-ca-rsa2048-g1"
  port                                = var.db_ingress_port
}

Steps to Reproduce

For a database with current ca_cert_identifier "rds-ca-2019" , update to "rds-ca-rsa2048-g1".

Debug Output

build 20-Sep-2023 10:59:14 �[0m�[0m�[1maws_s3_bucket_policy.flow_logs_policy: Modifying... [id=3556xxx-vpc-xxx]�[0m�[0m
build 20-Sep-2023 10:59:14 �[0m�[1mmodule.rds.aws_db_instance.db: Modifying... [id=db-YMBM3QDXXXX]�[0m�[0m
build 20-Sep-2023 10:59:14 �[0m�[1maws_s3_bucket_policy.flow_logs_policy: Modifications complete after 0s [id=3556xxx-vpc-0ebb17caaf2062b8c-flow-log-kerr-mg2-dev]�[0m
build 20-Sep-2023 10:59:24 �[0m�[1mmodule.rds.aws_db_instance.db: Still modifying... [id=db-YMBM3QDXXXX, 10s elapsed]�[0m�[0m
build 20-Sep-2023 10:59:34 �[0m�[1mmodule.rds.aws_db_instance.db: Still modifying... [id=db-YMBM3QDXXXX, 20s elapsed]�[0m�[0m
build 20-Sep-2023 10:59:44 �[0m�[1mmodule.rds.aws_db_instance.db: Still modifying... [id=db-YMBM3QDXXXX, 30s elapsed]�[0m�[0m
build 20-Sep-2023 10:59:54 �[0m�[1mmodule.rds.aws_db_instance.db: Still modifying... [id=db-YMBM3QDXXXX, 40s elapsed]�[0m�[0m
build 20-Sep-2023 11:00:04 �[0m�[1mmodule.rds.aws_db_instance.db: Still modifying... [id=db-YMBM3QDXXXX, 50s elapsed]�[0m�[0m
build 20-Sep-2023 11:00:14 �[0m�[1mmodule.rds.aws_db_instance.db: Still modifying... [id=db-YMBM3QDXXXX, 1m0s elapsed]�[0m�[0m
build 20-Sep-2023 11:00:24 �[0m�[1mmodule.rds.aws_db_instance.db: Still modifying... [id=db-YMBM3QDXXXX, 1m10s elapsed]�[0m�[0m
build 20-Sep-2023 11:00:34 �[0m�[1mmodule.rds.aws_db_instance.db: Still modifying... [id=db-YMBM3QDXXXX, 1m20s elapsed]�[0m�[0m
build 20-Sep-2023 11:00:35 �[0m�[1mmodule.rds.aws_db_instance.db: Modifications complete after 1m21s [id=db-YMBM3QDXXXX]�[0m

Panic Output

There is no panic.

Important Factoids

No response

References

No response

Would you like to implement a fix?

No

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[mgs255]

Also occurs with 5.17

[cg2v]

Rebooting isn't the same thing as having maintenance run. If you want the cert update to take effect before the next RDS maintenance cycle, you must use the apply_immediately argument in your aws_db_instance resource. (https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/db_instance#apply_immediately

[vbauchart]

Same problem but I can make it work by applying twice the same terraform configuration

1. At first terraform plan, it set the CA, but the paramater is not applied, it use the default CA instead

# aws_db_instance.backend will be created
  + resource "aws_db_instance" "backend" {
      + address                               = (known after apply)
      + allocated_storage                     = (known after apply)
      + apply_immediately                     = true
      + arn                                   = (known after apply)
      + auto_minor_version_upgrade            = true
      + availability_zone                     = (known after apply)
      + backup_retention_period               = 7
      + backup_target                         = (known after apply)
      + backup_window                         = "01:27-01:57"
      + ca_cert_identifier                    = "rds-ca-rsa2048-g1"

      [...]
      }

2. At second terraform plan, it detect the difference, and then apply it correctly (with apply_immediately =true)

  # module.rds_frepdb.aws_db_instance.backend will be updated in-place
  ~ resource "aws_db_instance" "backend" {
      ~ ca_cert_identifier                    = "rds-ca-2019" -> "rds-ca-rsa2048-g1"

    [...]
    }

This is a bug because the CA should be set at first run

[AlexAtkinson]

For the documentation, it would be supremely helpful if the arguments indicated which would be applied immediately regardless of the 'apply_immediately' setting, and which would be queued up for affect during the next maintenance window unless the apply_immediately setting was true. Though this is likely available in the aws docs... (confirmed)

For this bug itself, it also affects tf core 1.2.2, and aws provider 5.18.1.

NOTICE: For anyone who deploying RDS instances with blue_green_update enabled, with read replicas, and who needs to update away from 'rds-ca-2019', AWS will use the default certificate when deploying the new replicas during a blue-green update. This means that 'rds-ca-2019' will come back after such an update.
This can be overcome with the use of the modify-certificates command to set the default certificate for your AWS region. Here's a one liner to apply the change (to rds-ca-rsa2048-g1) for all regions.

for i in $(aws ec2 describe-regions --query "Regions[*].RegionName" --output text); do aws --region $i rds modify-certificates --certificate-identifier rds-ca-rsa2048-g1 ; done

Note that the modify-certificates doc states that this update is temporary, but it appears that "temporary" is until the certificate expires. Which for rsa2048 looks like this: "CustomerOverrideValidTill": "2061-05-19T20:04:06+00:00".

[mehiltne]

This is likely to become a more pressing matter as AWS approaches its cert transition:

Hello,

You are receiving this message because your AWS Account has one or more Amazon RDS, or Amazon Aurora database instances in the US-WEST-2 Region using an SSL/TLS Certificate that is expiring on August 22, 2024.

This is a follow-up notification for SSL/TLS CA certification expiration. If you believe you have already finished this work and still received this email it is likely because you created new instances using the 2019 CA. All newly created instances that don’t explicitly specify a different CA will use the 2019 CA until January 25, 2024 when the default will be switched to rds-ca-rsa2048-g1. For information on setting an account level CA override see the modify-certificates API documentation [1].

If your applications connect to these instances using the Secure Sockets Layer (SSL) or Transport Layer Security (TLS) protocol, you will need to take action before August 22, 2024 to prevent connectivity failures to your existing database instances. Even if you do not currently use SSL for your connections, you could still be affected if your databases server certificate expires, so we still recommend updating your CA.

... message continues ...

It looks like reminders are going out monthly, with the most recent one hitting affected accounts last Friday (4 days ago).

[tbugfinder]

I'm facing the same issue.
As the "default CA" expires mid 2024, it would be beneficial to get a fix.

[LasaleFamine]

Is this still a bug or we can use the ca_cert_identifier = "rds-ca-rsa2048-g1" without dealing with dirty states on terraform?

[dansd]

This is still an issue. I have multiple documentdb instances running with rds-ca-2019, and updating it to rds-ca-rsa2048-g1 via terraform apply does not have any effect. The instances keep the old certificate, and there is no planned maintenance visible (No Pending maintenance found.)

[BarnumD]

This is still an issue for me as well.
When applying ca_cert_identifier, it seems to take effect, but no maintenance is performed or scheduled. There is nothing to be done in the next maintenance schedule. The CA cert is still the old value.
I did also change apply_immediately to true and this caused the setting to take effect immediately.

I believe this is a bug, as changing ca_cert_identifier should at the least schedule the change for the next window.

[LasaleFamine]

So what's the best way to update the certificate on RDS in this situation? I mean, if we change the certificate with the CLI or from the AWS UI itself, will terraform see the change without broken the pipeline? This seems a bit critical, also because I suppose the certificate update will bring a downtime of the DB.

[Kammula280]

Hello
I'm on terraform-provider-aws version 4.27 and Terraform version 1.2.7

From what I have seen is when there is a change in ca_cert_identifier variable, Terraform apply works fine and queues it in pending modifications when apply_immediately is set to false which is working expected. Issue is that it's just not seen in AWS Console under pending modifications. If we use AWS CLI to list the pending modifications, we can see there is one when there is a change in ca_cert_identifier variable.
aws rds describe-db-instances --query 'DBInstances[*].{InstanceIdentifier:DBInstanceIdentifier, PendingModifiedValues:PendingModifiedValues}'

To update ca_cert_identifier

1. Just apply Terraform with a change in ca_cert_identifier and you can keep apply_immediately = false
2. Check if your certificate change is included in pending modifications using CLI command
   aws rds describe-db-instances --query 'DBInstances[*].{InstanceIdentifier:DBInstanceIdentifier, PendingModifiedValues:PendingModifiedValues}'
3. Apply your pending modifications using below CLI command
   aws rds modify-db-instance --db-instance-identifier <dbname> --apply-immediately
4. Check your database now which will have certificate with changed value.

Regarding downtime, it depends on the engine and engine version, you can use below CLI command with desired engine, engine version and check SupportsCertificateRotationWithoutRestart. If SupportsCertificateRotationWithoutRestart is true there will be no downtime.

aws rds describe-db-engine-versions --engine postgres --engine-version 14.7 --query 'DBEngineVersions[*].SupportsCertificateRotationWithoutRestart'

From now the terraform state will not have any issues and will match between actual and provided certificate.

Thank you!

[libsamek]

We are also experiencing a similar issue when setting up DocumentDB from scratch.

No matter what we set ca_cert_identifier parameter, the default is being used and at the time of writing this is rds-ca-rsa2048-g1.

Using apply_immediately=true helps, but the initial DocumentDB configuration is started with default CA and after a while certificate is switched to the actual value of ca_cert_identifier. This works but it is not ideal ...

EDIT: Is it possible to automate the above procedure mentioned by @Kammula280? To apply pending changes with Terraform and remove manual work.