bigquery table schema with field having missing mode attribute causes perma plan diff

[owennewo]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v0.12.26

Affected Resource(s)

• google_bigquery_table

Terraform Configuration Files

terraform {
  backend "gcs" {}
  required_version = "0.12.26"
}
provider "google" {
  version     = "v3.50.0"
  region      = var.region
  zone        = var.zone
}
provider "google-beta" {
  version     = "v3.50.0"
  region      = var.region
  zone        = var.zone
}

Debug Output

Panic Output

Expected Behavior

When a bigquery table schema has fields like this where mode of field is not specified:

schema = <<EOF
[  {
    "name": "PageNo",
    "type": "INTEGER"
  },
  {
    "name": "IngestTime",
    "type": "TIMESTAMP"
  }
]
EOF

After the google_bigquery_table resource is applied it should be possible to plan this resource and it should show zero change (nothing to do). i.e. It should not show perma change.

Actual Behavior

Terraform believes that the plan is requesting that "mode" : "NULLABLE" should be removed (note this mode is the default mode)

schema = <<EOF
[  {
   - "mode": "NULLABLE",
    "name": "PageNo",
    "type": "INTEGER"
  },
  {
   - "mode": "NULLABLE",
    "name": "IngestTime",
    "type": "TIMESTAMP"
  }
]
EOF

Not all field types have this issue. INTEGER and TIMESTAMP are affected but DATE is not

On its own this is a bit annoying, the workaround is to include the default mode in terraform.
However if JSON field ordering of schema has also changed (as in my case) it can show a massive delta (mostly field ordering changes) making the cause of this issue difficult to spot. Initially I thought #6803 (JSON schema diff suppresion) wasn't working but I believe if there is a change (e.g. due to this bug) then #6803 is unable to suppresses the field reordering parts. i.e. #6803 works great but only if there is no delta.

Steps to Reproduce

1. terraform apply
2. terraform plan

Important Factoids

This issue may be hard to reproduce. For instance it happens consistently in our dev environment but which has seen multiple schema changes applied to it but it did not happen in our QA environment which only had the final schema applied to it.

References

• #0000

[edwardmedia]

@owennewo can you share its debug log?

[jaycarlton]

@edwardmedia I can repro. My logs are DEBUG.
and TRACE.
In my case I've tried every combination of setting my fields to "NULLABLE", 'null', and just leaving the fields out. I still see this frequently in my workflow, where I create a number of tables from JSON files with a for loop.

[jaycarlton]

You can also observe a very similar phenomenon with the description field, which is optional, and cycles between "", null, and missing altogether in the input file. I've generally resolved to set this field every time to a nonempty string so it stays stable.

I want to say I've seen it on other optional fields as well, but I'm not sure.

[ScottSuarez]

@owennewo @jaycarlton

So I took a look at this resource and couldn't reproduce this. It does not seem like this mode:nullable reproduces on every configuration, and in my case I could not reproduce it. In our implementation we are simply surfacing the api get for this object and don't really do a whole lot on top of that. I went ahead and did a manual get on a configuration I deployed and confirmed that default wasn't getting inserted to the schema

This is the configuration I deployed

resource "google_bigquery_dataset" "test" {
	dataset_id = "tf_test_scoot"
}

resource "google_bigquery_table" "test" {
	table_id   = "tf_test_scooter"
	dataset_id = google_bigquery_dataset.test.dataset_id

	time_partitioning {
		type                     = "DAY"
		field                    = "IngestTime"
		require_partition_filter = true
	}
  schema = <<EOF
[  {
    "name": "PageNo",
    "type": "INTEGER"
  },
  {
    "name": "IngestTime",
    "type": "TIMESTAMP"
  }
]
EOF
}

and below is a scrubbed api get from the rest api

{
....
  "schema": {
    "fields": [
      {
        "name": "PageNo",
        "type": "INTEGER"
      },
      {
        "name": "IngestTime",
        "type": "TIMESTAMP"
      }
    ]
  },
  "timePartitioning": {
    "type": "DAY",
    "field": "IngestTime",
    "requirePartitionFilter": true
  },
  "numBytes": "0",
  "numLongTermBytes": "0",
  ...
}

Notice the "mode" : "NULLABLE" is not apart of the schema. I suspect this is some inconsistent api behavior. Could you supply me with your full (scrubbed) configuration so I can try to repo on my end?

I think the best solution here may be to file a bug with the api. We can also add support to suppress this diff in these scenarios but realistically we would want a configuration that we could add a testcase for. If I can't reproduce this I wouldn't want to add support that I can't maintain test coverage for.

[jaycarlton]

Most of our config is open source. Is there a tool for bundling up the whole thing and redacting it?

For repro, I'm beginning to think it takes several cycles to see the problem. I'm not sure of the order dependence but I could try with a fresh config.

[ScottSuarez]

We would just have to parse the json and do a manual comparison with these fields. The current function doing this comparison is located here. I'll take a stab at resolving this for you guys. @jaycarlton also gave me some of the example plans so hopefully I can get a repro on my side.

[owennewo]

@owennewo can you share its debug log?

bq_perma_diff_debug_plan.txt

Here is plan with TF_LOG=DEBUG

[ScottSuarez]

Checked in the fix for this.. thanks for your patience. Holidays held things up. Hope this will make your lives easier :)

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!