Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parameter engine_version ignored for aws_rds_cluster using engine aurora-postgresql #3557

Closed
spommerening opened this issue Feb 28, 2018 · 4 comments
Labels
Milestone

Comments

@spommerening
Copy link

@spommerening spommerening commented Feb 28, 2018

While trying to setup an Aurora PostgreSQL cluster it seems that the parameter engine_version is ignored. I was trying to setup a cluster with aurora-postgresql engine_version = "9.6.2" which is possible using aws_db_instance without further problems.

Terraform Version

Terraform v0.11.3
+ provider.aws v1.10.0

Affected Resource(s)

  • aws_rds_cluster

Terraform Configuration Files

resource "aws_rds_cluster_parameter_group" "dev" {
  name   = "aurora-pqsql-talend-dev"
  family = "aurora-postgresql9.6"
}

resource "aws_rds_cluster" "aurora_talend_dev" {
  cluster_identifier              = "aurora-cluster-psql-talend-dev"
  engine                          = "aurora-postgresql"
  engine_version                  = "9.6.2"
  skip_final_snapshot             = "false"
  final_snapshot_identifier       = "talend-dev-snapshot"
  vpc_security_group_ids          = ["${aws_security_group.talend_rds.id}"]
  master_username                 = "aurora"
  master_password                 = "<some-password>"
  db_subnet_group_name            = "${aws_db_subnet_group.talend_rds_priv.name}"
  db_cluster_parameter_group_name = "${aws_rds_cluster_parameter_group.dev.name}"
  preferred_backup_window         = "22:17-22:47"
  preferred_maintenance_window    = "mon:00:50-mon:01:20"
  backup_retention_period         = "7"
  apply_immediately               = "true"

  lifecycle {
    ignore_changes = ["master_password"]
  }
}

Output during create

aws_rds_cluster.aurora_talend_dev: Creating...
  apply_immediately:                 "" => "true"
  availability_zones.#:              "" => "<computed>"
  backup_retention_period:           "" => "7"
  cluster_identifier:                "" => "aurora-cluster-psql-talend-dev"
  cluster_identifier_prefix:         "" => "<computed>"
  cluster_members.#:                 "" => "<computed>"
  cluster_resource_id:               "" => "<computed>"
  database_name:                     "" => "<computed>"
  db_cluster_parameter_group_name:   "" => "aurora-pqsql-talend-dev"
  db_subnet_group_name:              "" => "talend-rds-priv-subnets"
  endpoint:                          "" => "<computed>"
  engine:                            "" => "aurora-postgresql"
  engine_version:                    "" => "9.6.2"
  final_snapshot_identifier:         "" => "talend-dev-snapshot"
  hosted_zone_id:                    "" => "<computed>"
  kms_key_id:                        "" => "<computed>"
  master_password:                   "<sensitive>" => "<sensitive>"
  master_username:                   "" => "aurora"
  port:                              "" => "<computed>"
  preferred_backup_window:           "" => "22:17-22:47"
  preferred_maintenance_window:      "" => "mon:00:50-mon:01:20"
  reader_endpoint:                   "" => "<computed>"
  skip_final_snapshot:               "" => "false"
  storage_encrypted:                 "" => "false"
  vpc_security_group_ids.#:          "" => "1"
  vpc_security_group_ids.3121866956: "" => "sg-733ceb1e"

Expected Behavior

I expect an RDS cluster postgresql 9.6.2 to be created.

Actual Behavior

An RDS cluster postgresql 9.6.3 was created:

$ aws rds describe-db-clusters --db-cluster-identifier aurora-cluster-psql-talend-dev
{
    "DBClusters": [
        {
            "MasterUsername": "aurora",
            "ReaderEndpoint": "aurora-cluster-psql-talend-dev.cluster-ro-cq2hxhgxq9pi.eu-central-1.rds.amazonaws.com",
            "ReadReplicaIdentifiers": [],
            "VpcSecurityGroups": [
                {
                    "Status": "active",
                    "VpcSecurityGroupId": "sg-733ceb1e"
                }
            ],
            "HostedZoneId": "Z1RLNUO7B9Q6NB",
            "Status": "available",
            "MultiAZ": false,
            "LatestRestorableTime": "2018-02-28T09:26:04.935Z",
            "PreferredBackupWindow": "22:17-22:47",
            "DBSubnetGroup": "talend-rds-priv-subnets",
            "AllocatedStorage": 1,
            "BackupRetentionPeriod": 7,
            "PreferredMaintenanceWindow": "mon:00:50-mon:01:20",
            "Engine": "aurora-postgresql",
            "Endpoint": "aurora-cluster-psql-talend-dev.cluster-cq2hxhgxq9pi.eu-central-1.rds.amazonaws.com",
            "EarliestRestorableTime": "2018-02-28T09:26:04.935Z",
            "IAMDatabaseAuthenticationEnabled": false,
            "ClusterCreateTime": "2018-02-28T09:25:10.207Z",
            "EngineVersion": "9.6.3",
            "DBClusterIdentifier": "aurora-cluster-psql-talend-dev",
            "DbClusterResourceId": "cluster-HUN6OWEDWELZB4V3UGKNPG7G3U",
            "DBClusterMembers": [],
            "DBClusterArn": "arn:aws:rds:eu-central-1:199636132489:cluster:aurora-cluster-psql-talend-dev",
            "StorageEncrypted": false,
            "AssociatedRoles": [],
            "DBClusterParameterGroup": "aurora-pqsql-talend-dev",
            "AvailabilityZones": [
                "eu-central-1a",
                "eu-central-1b",
                "eu-central-1c"
            ],
            "Port": 5432
        }
    ]
}

Repeating terraform plan will endlessly rebuild the same cluster over and over again:

Terraform will perform the following actions:

-/+ aws_rds_cluster.aurora_talend_dev (new resource required)
      id:                                "aurora-cluster-psql-talend-dev" => <computed> (forces new resource)
      apply_immediately:                 "true" => "true"
      availability_zones.#:              "3" => <computed>
      backup_retention_period:           "7" => "7"
      cluster_identifier:                "aurora-cluster-psql-talend-dev" => "aurora-cluster-psql-talend-dev"
      cluster_identifier_prefix:         "" => <computed>
      cluster_members.#:                 "0" => <computed>
      cluster_resource_id:               "cluster-HUN6OWEDWELZB4V3UGKNPG7G3U" => <computed>
      database_name:                     "" => <computed>
      db_cluster_parameter_group_name:   "aurora-pqsql-talend-dev" => "aurora-pqsql-talend-dev"
      db_subnet_group_name:              "talend-rds-priv-subnets" => "talend-rds-priv-subnets"
      endpoint:                          "aurora-cluster-psql-talend-dev.cluster-cq2hxhgxq9pi.eu-central-1.rds.amazonaws.com" => <computed>
      engine:                            "aurora-postgresql" => "aurora-postgresql"
      engine_version:                    "9.6.3" => "9.6.2" (forces new resource)
      final_snapshot_identifier:         "talend-dev-snapshot" => "talend-dev-snapshot"
      hosted_zone_id:                    "Z1RLNUO7B9Q6NB" => <computed>
      kms_key_id:                        "" => <computed>
      master_password:                   <sensitive> => <sensitive> (attribute changed)
      master_username:                   "aurora" => "aurora"
      port:                              "5432" => <computed>
      preferred_backup_window:           "22:17-22:47" => "22:17-22:47"
      preferred_maintenance_window:      "mon:00:50-mon:01:20" => "mon:00:50-mon:01:20"
      reader_endpoint:                   "aurora-cluster-psql-talend-dev.cluster-ro-cq2hxhgxq9pi.eu-central-1.rds.amazonaws.com" => <computed>
      skip_final_snapshot:               "false" => "false"
      storage_encrypted:                 "false" => "false"
      vpc_security_group_ids.#:          "1" => "1"
      vpc_security_group_ids.3121866956: "sg-733ceb1e" => "sg-733ceb1e"
@ageekymonk

This comment has been minimized.

Copy link

@ageekymonk ageekymonk commented Mar 1, 2018

@spommerening For aurora there is no option for selecting specific version of postgres. Thats why it is not working.
But you have uncovered another bug aws_rds_cluster should not have engine_version as it is not supported. We should remove it, that way we dont actually add one causing rebuild of apply everytime if it does not match.

@gevansBigG

This comment has been minimized.

Copy link

@gevansBigG gevansBigG commented Apr 6, 2018

The version of postgres on rds_cluster can be specified. You can change the version on create or on modify via console.
On create:
image

On modify:
image

the CLI create_db_cluster also seems to support the --engine-version parameter:
https://docs.aws.amazon.com/cli/latest/reference/rds/create-db-cluster.html
image

the API docs also suggest this param is valid:
https://docs.aws.amazon.com/AmazonRDS/latest/APIReference/API_CreateDBCluster.html

I've run into this same issue as the OP trying to create clusters with 9.6.6. If I do not pass an engine_version to the aws_rds_cluster and aws_rds_cluster_instance resources, I get working builds on 9.6.3. However, if I try to use 9.6.6, my apply fails with this output:

* module.gevans-test-terraform-aurora-cluster.aws_rds_cluster_instance.aurora_instance[0]: 1 error(s) occurred:

* aws_rds_cluster_instance.aurora_instance.0: InvalidParameterCombination: The engine version that you requested for your DB instance (9.6.6) does not match the engine version of your DB cluster (9.6.3).
	status code: 400, request id: 33687814-56d6-4030-bf76-9063fddcbbf8

My plan that generated the error did show 9.6.6 as the engine_version for the cluster. This was tested with the 1.13.0 provider.

module.gevans-test-terraform-aurora-cluster.aws_rds_cluster.main_aurora_cluster: Creating...
  apply_immediately:                 "" => "true"
  availability_zones.#:              "" => "3"
  availability_zones.2050015877:     "" => "us-west-2c"
  availability_zones.221770259:      "" => "us-west-2b"
  availability_zones.2487133097:     "" => "us-west-2a"
  backup_retention_period:           "" => "1"
  cluster_identifier:                "" => "ge-product-ge-service-development"
  cluster_identifier_prefix:         "" => "<computed>"
  cluster_members.#:                 "" => "<computed>"
  cluster_resource_id:               "" => "<computed>"
  database_name:                     "" => "ge_test"
  db_cluster_parameter_group_name:   "" => "<computed>"
  db_subnet_group_name:              "" => "ge-product-ge-service-development_aurora_db_subnet_group"
  endpoint:                          "" => "<computed>"
  engine:                            "" => "aurora-postgresql"
  engine_version:                    "" => "9.6.6"
....
@bflad bflad added this to the v1.14.1 milestone Apr 10, 2018
@bflad

This comment has been minimized.

Copy link
Contributor

@bflad bflad commented Apr 10, 2018

This fix for not properly handling the engine_version attribute on normal creation of aws_rds_cluster resources (#4139) has been merged into master and will release with v1.14.1 of the AWS provider, likely tomorrow.

@bflad bflad closed this Apr 10, 2018
@bflad

This comment has been minimized.

Copy link
Contributor

@bflad bflad commented Apr 11, 2018

This has been released in version 1.14.1 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.