Parameter engine_version ignored for aws_rds_cluster using engine aurora-postgresql

[spommerening]

While trying to setup an Aurora PostgreSQL cluster it seems that the parameter engine_version is ignored. I was trying to setup a cluster with aurora-postgresql engine_version = "9.6.2" which is possible using aws_db_instance without further problems.

Terraform Version

Terraform v0.11.3
+ provider.aws v1.10.0

Affected Resource(s)

• aws_rds_cluster

Terraform Configuration Files

resource "aws_rds_cluster_parameter_group" "dev" {
  name   = "aurora-pqsql-talend-dev"
  family = "aurora-postgresql9.6"
}

resource "aws_rds_cluster" "aurora_talend_dev" {
  cluster_identifier              = "aurora-cluster-psql-talend-dev"
  engine                          = "aurora-postgresql"
  engine_version                  = "9.6.2"
  skip_final_snapshot             = "false"
  final_snapshot_identifier       = "talend-dev-snapshot"
  vpc_security_group_ids          = ["${aws_security_group.talend_rds.id}"]
  master_username                 = "aurora"
  master_password                 = "<some-password>"
  db_subnet_group_name            = "${aws_db_subnet_group.talend_rds_priv.name}"
  db_cluster_parameter_group_name = "${aws_rds_cluster_parameter_group.dev.name}"
  preferred_backup_window         = "22:17-22:47"
  preferred_maintenance_window    = "mon:00:50-mon:01:20"
  backup_retention_period         = "7"
  apply_immediately               = "true"

  lifecycle {
    ignore_changes = ["master_password"]
  }
}

Output during create

aws_rds_cluster.aurora_talend_dev: Creating...
  apply_immediately:                 "" => "true"
  availability_zones.#:              "" => "<computed>"
  backup_retention_period:           "" => "7"
  cluster_identifier:                "" => "aurora-cluster-psql-talend-dev"
  cluster_identifier_prefix:         "" => "<computed>"
  cluster_members.#:                 "" => "<computed>"
  cluster_resource_id:               "" => "<computed>"
  database_name:                     "" => "<computed>"
  db_cluster_parameter_group_name:   "" => "aurora-pqsql-talend-dev"
  db_subnet_group_name:              "" => "talend-rds-priv-subnets"
  endpoint:                          "" => "<computed>"
  engine:                            "" => "aurora-postgresql"
  engine_version:                    "" => "9.6.2"
  final_snapshot_identifier:         "" => "talend-dev-snapshot"
  hosted_zone_id:                    "" => "<computed>"
  kms_key_id:                        "" => "<computed>"
  master_password:                   "<sensitive>" => "<sensitive>"
  master_username:                   "" => "aurora"
  port:                              "" => "<computed>"
  preferred_backup_window:           "" => "22:17-22:47"
  preferred_maintenance_window:      "" => "mon:00:50-mon:01:20"
  reader_endpoint:                   "" => "<computed>"
  skip_final_snapshot:               "" => "false"
  storage_encrypted:                 "" => "false"
  vpc_security_group_ids.#:          "" => "1"
  vpc_security_group_ids.3121866956: "" => "sg-733ceb1e"

Expected Behavior

I expect an RDS cluster postgresql 9.6.2 to be created.

Actual Behavior

An RDS cluster postgresql 9.6.3 was created:

$ aws rds describe-db-clusters --db-cluster-identifier aurora-cluster-psql-talend-dev
{
    "DBClusters": [
        {
            "MasterUsername": "aurora",
            "ReaderEndpoint": "aurora-cluster-psql-talend-dev.cluster-ro-cq2hxhgxq9pi.eu-central-1.rds.amazonaws.com",
            "ReadReplicaIdentifiers": [],
            "VpcSecurityGroups": [
                {
                    "Status": "active",
                    "VpcSecurityGroupId": "sg-733ceb1e"
                }
            ],
            "HostedZoneId": "Z1RLNUO7B9Q6NB",
            "Status": "available",
            "MultiAZ": false,
            "LatestRestorableTime": "2018-02-28T09:26:04.935Z",
            "PreferredBackupWindow": "22:17-22:47",
            "DBSubnetGroup": "talend-rds-priv-subnets",
            "AllocatedStorage": 1,
            "BackupRetentionPeriod": 7,
            "PreferredMaintenanceWindow": "mon:00:50-mon:01:20",
            "Engine": "aurora-postgresql",
            "Endpoint": "aurora-cluster-psql-talend-dev.cluster-cq2hxhgxq9pi.eu-central-1.rds.amazonaws.com",
            "EarliestRestorableTime": "2018-02-28T09:26:04.935Z",
            "IAMDatabaseAuthenticationEnabled": false,
            "ClusterCreateTime": "2018-02-28T09:25:10.207Z",
            "EngineVersion": "9.6.3",
            "DBClusterIdentifier": "aurora-cluster-psql-talend-dev",
            "DbClusterResourceId": "cluster-HUN6OWEDWELZB4V3UGKNPG7G3U",
            "DBClusterMembers": [],
            "DBClusterArn": "arn:aws:rds:eu-central-1:199636132489:cluster:aurora-cluster-psql-talend-dev",
            "StorageEncrypted": false,
            "AssociatedRoles": [],
            "DBClusterParameterGroup": "aurora-pqsql-talend-dev",
            "AvailabilityZones": [
                "eu-central-1a",
                "eu-central-1b",
                "eu-central-1c"
            ],
            "Port": 5432
        }
    ]
}

Repeating terraform plan will endlessly rebuild the same cluster over and over again:

Terraform will perform the following actions:

-/+ aws_rds_cluster.aurora_talend_dev (new resource required)
      id:                                "aurora-cluster-psql-talend-dev" => <computed> (forces new resource)
      apply_immediately:                 "true" => "true"
      availability_zones.#:              "3" => <computed>
      backup_retention_period:           "7" => "7"
      cluster_identifier:                "aurora-cluster-psql-talend-dev" => "aurora-cluster-psql-talend-dev"
      cluster_identifier_prefix:         "" => <computed>
      cluster_members.#:                 "0" => <computed>
      cluster_resource_id:               "cluster-HUN6OWEDWELZB4V3UGKNPG7G3U" => <computed>
      database_name:                     "" => <computed>
      db_cluster_parameter_group_name:   "aurora-pqsql-talend-dev" => "aurora-pqsql-talend-dev"
      db_subnet_group_name:              "talend-rds-priv-subnets" => "talend-rds-priv-subnets"
      endpoint:                          "aurora-cluster-psql-talend-dev.cluster-cq2hxhgxq9pi.eu-central-1.rds.amazonaws.com" => <computed>
      engine:                            "aurora-postgresql" => "aurora-postgresql"
      engine_version:                    "9.6.3" => "9.6.2" (forces new resource)
      final_snapshot_identifier:         "talend-dev-snapshot" => "talend-dev-snapshot"
      hosted_zone_id:                    "Z1RLNUO7B9Q6NB" => <computed>
      kms_key_id:                        "" => <computed>
      master_password:                   <sensitive> => <sensitive> (attribute changed)
      master_username:                   "aurora" => "aurora"
      port:                              "5432" => <computed>
      preferred_backup_window:           "22:17-22:47" => "22:17-22:47"
      preferred_maintenance_window:      "mon:00:50-mon:01:20" => "mon:00:50-mon:01:20"
      reader_endpoint:                   "aurora-cluster-psql-talend-dev.cluster-ro-cq2hxhgxq9pi.eu-central-1.rds.amazonaws.com" => <computed>
      skip_final_snapshot:               "false" => "false"
      storage_encrypted:                 "false" => "false"
      vpc_security_group_ids.#:          "1" => "1"
      vpc_security_group_ids.3121866956: "sg-733ceb1e" => "sg-733ceb1e"

[ageekymonk]

@spommerening For aurora there is no option for selecting specific version of postgres. Thats why it is not working.
But you have uncovered another bug aws_rds_cluster should not have engine_version as it is not supported. We should remove it, that way we dont actually add one causing rebuild of apply everytime if it does not match.

[gevansBigG]

The version of postgres on rds_cluster can be specified. You can change the version on create or on modify via console.
On create:

On modify:

the CLI create_db_cluster also seems to support the --engine-version parameter:
https://docs.aws.amazon.com/cli/latest/reference/rds/create-db-cluster.html

the API docs also suggest this param is valid:
https://docs.aws.amazon.com/AmazonRDS/latest/APIReference/API_CreateDBCluster.html

I've run into this same issue as the OP trying to create clusters with 9.6.6. If I do not pass an engine_version to the aws_rds_cluster and aws_rds_cluster_instance resources, I get working builds on 9.6.3. However, if I try to use 9.6.6, my apply fails with this output:

* module.gevans-test-terraform-aurora-cluster.aws_rds_cluster_instance.aurora_instance[0]: 1 error(s) occurred:

* aws_rds_cluster_instance.aurora_instance.0: InvalidParameterCombination: The engine version that you requested for your DB instance (9.6.6) does not match the engine version of your DB cluster (9.6.3).
	status code: 400, request id: 33687814-56d6-4030-bf76-9063fddcbbf8

My plan that generated the error did show 9.6.6 as the engine_version for the cluster. This was tested with the 1.13.0 provider.

module.gevans-test-terraform-aurora-cluster.aws_rds_cluster.main_aurora_cluster: Creating...
  apply_immediately:                 "" => "true"
  availability_zones.#:              "" => "3"
  availability_zones.2050015877:     "" => "us-west-2c"
  availability_zones.221770259:      "" => "us-west-2b"
  availability_zones.2487133097:     "" => "us-west-2a"
  backup_retention_period:           "" => "1"
  cluster_identifier:                "" => "ge-product-ge-service-development"
  cluster_identifier_prefix:         "" => "<computed>"
  cluster_members.#:                 "" => "<computed>"
  cluster_resource_id:               "" => "<computed>"
  database_name:                     "" => "ge_test"
  db_cluster_parameter_group_name:   "" => "<computed>"
  db_subnet_group_name:              "" => "ge-product-ge-service-development_aurora_db_subnet_group"
  endpoint:                          "" => "<computed>"
  engine:                            "" => "aurora-postgresql"
  engine_version:                    "" => "9.6.6"
....

[bflad]

This fix for not properly handling the engine_version attribute on normal creation of aws_rds_cluster resources (#4139) has been merged into master and will release with v1.14.1 of the AWS provider, likely tomorrow.

[bflad]

This has been released in version 1.14.1 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!