aws_glue_catalog_table is not changed when aws_glue_schema dependency is updated

[schmoesta]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.2.4
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v4.22.0

Affected Resource(s)

• aws_glue_schema
• aws_glue_catalog_table

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.22.0"
    }
  }
}

provider "aws" {
  region = "eu-central-1"
}

resource "aws_glue_catalog_database" "streams" {
  name = "streams"
}

resource "aws_glue_registry" "streams" {
  registry_name = "streams"
}

resource "aws_glue_schema" "data_stream" {
  schema_name   = "data_stream"
  data_format   = "AVRO"
  compatibility = "FORWARD"
  schema_definition = jsonencode({
    "namespace" : "my.schemas",
    "type" : "record",
    "name" : "Schema",
    "fields" : [
      {
        "name" : "field_1",
        "type" : "string"
      }
    ]
  })
  depends_on = [
    aws_glue_registry.streams
  ]
  registry_arn = aws_glue_registry.streams.arn
}

resource "aws_kinesis_stream" "data_stream" {
  name        = "data_stream"
  shard_count = 1
}

resource "aws_glue_catalog_table" "data_stream" {
  name = "data_stream"
  depends_on = [
    aws_glue_catalog_database.streams,
    aws_glue_schema.data_stream,
    aws_kinesis_stream.data_stream,
  ]
  database_name = aws_glue_catalog_database.streams.name
  parameters = {
    "classification" = "avro"
  }
  storage_descriptor {
    location      = aws_kinesis_stream.data_stream.name
    input_format  = "org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat"
    output_format = "org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat"
    compressed    = false
    parameters = {
      "streamARN"  = aws_kinesis_stream.data_stream.arn
      "typeOfData" = "kinesis"
    }
    schema_reference {
      schema_id {
        schema_arn = aws_glue_schema.data_stream.arn
      }
      schema_version_number = aws_glue_schema.data_stream.latest_schema_version
    }
  }
}

Expected Behavior

If I change an aws_glue_schema, an aws_glue_catalog_table depending on that schema should be updated as well.

Actual Behavior

An initial apply of the example above works fine. If I now change the schema of the aws_glue_schema resource and apply again, a new version of that schema will be registered in the Glue Schema Registry - so far so good. However, the aws_glue_catalog_table, which is set up to use the latest_schema_version is not updated. If I run terraform apply again, it picks up the new version of the schema and updates the table.

Steps to Reproduce

1. terraform apply - an initial version of the schema and table are created
2. modify the schema, e.g. by adding an optional new field field_2:

...
schema_definition = jsonencode({
    "namespace" : "my.schemas",
    "type" : "record",
    "name" : "Schema",
    "fields" : [
      {
        "name" : "field_1",
        "type" : "string"
      },
      {
        "name" : "field_2",
        "type" : ["string", "null"]
      }
    ]
  })
...

3. terraform apply - the aws_glue_schema is modified and a new version of the schema is registered.
4. terraform apply - now aws_glue_catalog_table is updated accordingly

Important Factoids

I'm quite new to Terraform and hope I'm not doing something completely stupid here.

References

I've been looking around for other issues where people had to run terraform apply twice to achieve what they wanted, but didn't quite find what I was looking for:

• #12572
• #745

[weaversam8]

I'm also encountering this issue.

The aws_glue_schema resource has a next_schema_version attribute that looks like it might be able to be used here to automatically upgrade the schema version on the aws_glue_table resource, but I can't figure out how to only trigger an update on that argument when the glue_schema is changed.

[ashish-mokashe]

I am also having same issue.

Table doesn't update even after changing any properties

[miky-schreiber]

We're also facing this issue. If it can be fixed it would be great and very helpful.

[tde-desc]

Same issue here.

[PF1o1]

Same issue here. Can someone please review this bug?