Terraform BigQuery Table Hive partitioning support #3335

ffung · 2020-04-03T07:21:01Z

Fixes: hashicorp/terraform-provider-google#5664
As of March 2, range partioning / hive partitioning is GA, see https://cloud.google.com/bigquery/docs/release-notes.
Note: Doesn't support require_partition_filter attribute as this isn't available from the used BigQuery SDK.

Release Note Template for Downstream PRs (will be copied)

bigquery: Added support for `google_bigquery_table` `hive_partitioning_options`

bigquery: Added `google_bigquery_table` `range_partitioning` to GA

modular-magician · 2020-04-03T07:22:28Z

Hello! I am a robot who works on Magic Modules PRs.

I have detected that you are a community contributor, so your PR will be assigned to someone with a commit-bit on this repo for initial review.

Thanks for your contribution! A human will be with you soon.

@slevenick, please review this PR or find an appropriate assignee.

modular-magician · 2020-04-03T07:26:10Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 318 insertions(+), 3 deletions(-))
Terraform Beta: Diff ( 3 files changed, 152 insertions(+), 5 deletions(-))

modular-magician · 2020-04-03T07:26:16Z

Oops! It looks like you're using an unknown release-note type in your changelog entries:

REPLACEME

Please only use the types listed in https://github.com/GoogleCloudPlatform/magic-modules/blob/master/.ci/RELEASE_NOTES_GUIDE.md.

ffung · 2020-04-03T13:47:41Z

± env |grep GOOG
GOOGLE_REGION=us-central1
GOOGLE_ZONE=us-central1-a

± make testacc TEST=./google TESTARGS='-run=TestAccBigQueryTable_Hive'

==> Checking source code against gofmt...
==> Checking that code complies with gofmt requirements...
TF_ACC=1 TF_SCHEMA_PANIC_ON_ERROR=1 go test ./google -v -run=TestAccBigQueryTable_Hive -timeout 240m -ldflags="-X=github.com/terraform-providers/terraform-provider-google/version.ProviderVersion=acc"
=== RUN   TestAccBigQueryTable_HivePartitioning
=== PAUSE TestAccBigQueryTable_HivePartitioning
=== CONT  TestAccBigQueryTable_HivePartitioning
--- PASS: TestAccBigQueryTable_HivePartitioning (8.96s)
PASS
ok  	github.com/terraform-providers/terraform-provider-google/google	(cached)

slevenick

Thanks for the contribution!

Sorry it took so long to review, but this all looks great. Just a few notes on docs/tests. When you push a new commit it should successfully complete the checks as I standardized the release note comment

third_party/terraform/website/docs/r/bigquery_table.html.markdown

third_party/terraform/tests/resource_bigquery_table_test.go

modular-magician · 2020-04-16T11:40:40Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 321 insertions(+), 3 deletions(-))
Terraform Beta: Diff ( 3 files changed, 155 insertions(+), 5 deletions(-))

modular-magician · 2020-04-16T11:42:51Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 321 insertions(+), 3 deletions(-))
Terraform Beta: Diff ( 3 files changed, 155 insertions(+), 5 deletions(-))

slevenick

Minor issues and this will likely require a rebase due to an unrelated PR that changed test structure.

If you can do the rebase I'll get this merged in, looks good!

third_party/terraform/tests/resource_bigquery_table_test.go

modular-magician · 2020-04-17T08:18:06Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 321 insertions(+), 3 deletions(-))
Terraform Beta: Diff ( 3 files changed, 155 insertions(+), 5 deletions(-))

modular-magician · 2020-04-17T08:22:16Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 321 insertions(+), 3 deletions(-))
Terraform Beta: Diff ( 3 files changed, 155 insertions(+), 5 deletions(-))

modular-magician · 2020-05-06T14:37:36Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 327 insertions(+), 7 deletions(-))
Terraform Beta: Diff ( 3 files changed, 161 insertions(+), 9 deletions(-))

ffung · 2020-05-20T06:58:24Z

@slevenick I think I resolved all the issues, is there anything I still can do for this PR?

slevenick · 2020-05-20T16:15:56Z

@slevenick I think I resolved all the issues, is there anything I still can do for this PR?

Ah, sorry lost track of this. Looks like there are conflicts now with master, could you rebase on master and I can run tests and get this merged?

modular-magician · 2020-05-21T08:22:37Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 327 insertions(+), 7 deletions(-))
Terraform Beta: Diff ( 3 files changed, 161 insertions(+), 9 deletions(-))

ffung · 2020-05-21T08:26:38Z

@slevenick done.I cannot see why the build fails, but let me know if I can help address the issue.

slevenick · 2020-05-21T18:36:12Z

@slevenick done.I cannot see why the build fails, but let me know if I can help address the issue.

Looks like the files aren't formatted correctly:

gofmt needs running on the following files:
./google/resource_bigquery_table_test.go
./google/resource_bigquery_table.go```

You can generate the downstream files, run `gofmt` on them and then copy them back into Magic Modules to fix this

modular-magician · 2020-05-21T19:41:35Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 321 insertions(+), 3 deletions(-))
Terraform Beta: Diff ( 3 files changed, 155 insertions(+), 5 deletions(-))

ffung · 2020-05-21T19:45:50Z

@slevenick ran gofmt, build still fails though

slevenick · 2020-05-21T19:57:59Z

@slevenick ran gofmt, build still fails though

Looks like the problem is now:

google/resource_bigquery_table_test.go:69:30: not enough arguments in call to testBucketName
	have ()
	want (*testing.T)

You can run make test in the generated provider to make sure everything passes

modular-magician · 2020-05-22T20:05:37Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 321 insertions(+), 3 deletions(-))
Terraform Beta: Diff ( 3 files changed, 155 insertions(+), 5 deletions(-))

ffung · 2020-05-22T20:12:21Z

@slevenick ah sorry that was sloppy, tests are running succesfully now locally, but the builds still seems to fail.

slevenick · 2020-05-26T16:06:35Z

Sorry the process is a little rough around the edges. Looks like we enforce that gofmt must be run with a specific flag 😞

Step #22 - "tpgb-test": google-beta/resource_bigquery_table.go:336: File is not `gofmt`-ed with `-s` (gofmt)
Step #22 - "tpgb-test": 			"range_partitioning": &schema.Schema{```

modular-magician · 2020-05-27T12:24:45Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 320 insertions(+), 2 deletions(-))
Terraform Beta: Diff ( 3 files changed, 152 insertions(+), 2 deletions(-))

ffung · 2020-05-27T12:27:28Z

@slevenick that should fix it;-)

slevenick

Thanks for the addition!

LoekL · 2020-06-24T09:43:55Z

FYI I tried using this today, no dice. The table does get created:

But there's no sign of the partitions in the table schema:

If I however run:

bq mkdef \
  --autodetect \
  --ignore_unknown_values \
  --source_format=NEWLINE_DELIMITED_JSON \
  --hive_partitioning_mode=CUSTOM \
  --hive_partitioning_source_uri_prefix=gs://anhistous-metonymic-7578834-analytics-staging-batched/data_type=jsonl/event_schema=playfab/{event_category:STRING}/{event_environment:STRING}/{event_date:DATE}/{event_hour:STRING}/{event_minute:STRING} \
  gs://anhistous-metonymic-7578834-analytics-staging-batched/data_type=jsonl/event_schema=playfab/\* \
  AnalyticsEnvironment:STRING,PlayFabEnvironment:STRING,SourceType:STRING,Source:STRING,EventNamespace:STRING,TitleId:STRING,GroupBatchId:STRING,BatchId:STRING,EventId:STRING,EventName:STRING,EntityType:STRING,EntityId:STRING,Timestamp:TIMESTAMP,ReceivedTimestamp:TIMESTAMP,BatchedTimestamp:TIMESTAMP,BatchJobName:STRING,EventAttributes:STRING \
  > /Users/loek/Desktop/events_playfab_staging_hive_partitioned_batched

Followed by:

bq mk --table --location=US --external_table_definition=/Users/loek/Desktop/events_playfab_staging_hive_partitioned_batched external.events_playfab_hive_partitioned_batched

It does work:

&

The only difference within the BQ UI I can see is that when going via the bq CLI, I don't see Compression = GZIP in the table details. Am using Terraform v0.12.18 & provider.google v3.26.0.

slevenick · 2020-06-24T16:42:40Z

@LoekL can you create a new issue on https://github.com/terraform-providers/terraform-provider-google with steps to reproduce this problem so that we can look into this?

ffung · 2020-06-25T07:45:57Z

yes @LoekL please raise an issue, I cannot reproduce your problem. If I apply this, hive partitioning works just fine.

resource "google_bigquery_dataset" "test" {
  dataset_id                  = "foo"
  friendly_name               = "test"
  description                 = "This is a test description"
  location                    = "europe-west3"
  default_table_expiration_ms = 3600000

  labels = {
    env = "default"
  }
}

resource "google_bigquery_table" "test" {
  dataset_id = google_bigquery_dataset.test.dataset_id
  table_id   = "bar"

  external_data_configuration {
    source_format = "PARQUET"
    autodetect = true
    source_uris= ["gs://project-2resdfdsafsdf43/test/*"]

    hive_partitioning_options {
      mode = "CUSTOM"
      source_uri_prefix = "gs://project-2resdfdsafsdf43/test/{part:STRING}/{dt:DATE}"
    }
  }
}

LoekL · 2020-06-25T10:41:28Z

@slevenick @ffung - see: hashicorp/terraform-provider-google#6693

googlebot added the cla: yes label Apr 3, 2020

modular-magician added the community label Apr 3, 2020

danawillow requested a review from slevenick April 14, 2020 23:43

slevenick requested changes Apr 15, 2020

View reviewed changes

slevenick requested changes Apr 16, 2020

View reviewed changes

third_party/terraform/tests/resource_bigquery_table_test.go Outdated Show resolved Hide resolved

third_party/terraform/tests/resource_bigquery_table_test.go Outdated Show resolved Hide resolved

third_party/terraform/tests/resource_bigquery_table_test.go Outdated Show resolved Hide resolved

ffung force-pushed the tf_bigquery_hive_partitioning_5664 branch from 0db16d3 to da02a5e Compare April 17, 2020 08:13

ffung force-pushed the tf_bigquery_hive_partitioning_5664 branch from de0518e to cb238ce Compare May 6, 2020 14:31

ffung requested a review from slevenick May 6, 2020 14:38

ffung added 5 commits May 21, 2020 10:15

range partitioning for BigQuery is GA

38fa232

add hive partitioning options to google_bigquery_table

54a3876

improve on formatting of bigquery table hive partitioning options

43d8a60

correct indenting on resource_bigquery_table_test.go

3f8e2a3

minor fix on the documentation of bigquery table

af88aa7

ffung force-pushed the tf_bigquery_hive_partitioning_5664 branch from cb238ce to cdfef07 Compare May 21, 2020 08:16

align bigquery table test with upstream changes

0a970c7

ffung force-pushed the tf_bigquery_hive_partitioning_5664 branch from 75807a4 to c101365 Compare May 22, 2020 19:59

gofmt on resource_bigquery_table, resource_bigquery_table_test.go

5d55d27

ffung force-pushed the tf_bigquery_hive_partitioning_5664 branch from c101365 to 5d55d27 Compare May 27, 2020 12:18

slevenick approved these changes May 29, 2020

View reviewed changes

slevenick merged commit 43bf665 into GoogleCloudPlatform:master May 29, 2020

This was referenced May 29, 2020

Terraform BigQuery Table Hive partitioning support hashicorp/terraform-provider-google#6488

Merged

Terraform BigQuery Table Hive partitioning support hashicorp/terraform-provider-google-beta#2121

Merged

ffung deleted the tf_bigquery_hive_partitioning_5664 branch June 5, 2020 07:19

LoekL mentioned this pull request Jun 25, 2020

BigQuery Table Hive Partitioning not Working with Explicit Schema (autodetect=false) hashicorp/terraform-provider-google#6693

Closed

Terraform BigQuery Table Hive partitioning support #3335

Terraform BigQuery Table Hive partitioning support #3335

Uh oh!

Conversation

ffung commented Apr 3, 2020 • edited by slevenick Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

modular-magician commented Apr 3, 2020

Uh oh!

modular-magician commented Apr 3, 2020

Diff report:

Uh oh!

modular-magician commented Apr 3, 2020

Uh oh!

ffung commented Apr 3, 2020

Uh oh!

slevenick left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

modular-magician commented Apr 16, 2020

Diff report:

Uh oh!

modular-magician commented Apr 16, 2020

Diff report:

Uh oh!

slevenick left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

modular-magician commented Apr 17, 2020

Diff report:

Uh oh!

modular-magician commented Apr 17, 2020

Diff report:

Uh oh!

modular-magician commented May 6, 2020

Diff report:

Uh oh!

ffung commented May 20, 2020

Uh oh!

slevenick commented May 20, 2020

Uh oh!

modular-magician commented May 21, 2020

Diff report:

Uh oh!

ffung commented May 21, 2020

Uh oh!

slevenick commented May 21, 2020

Uh oh!

modular-magician commented May 21, 2020

Diff report:

Uh oh!

ffung commented May 21, 2020

Uh oh!

slevenick commented May 21, 2020

Uh oh!

modular-magician commented May 22, 2020

Diff report:

Uh oh!

ffung commented May 22, 2020

Uh oh!

slevenick commented May 26, 2020

Uh oh!

modular-magician commented May 27, 2020

Diff report:

Uh oh!

ffung commented May 27, 2020

Uh oh!

slevenick left a comment

Choose a reason for hiding this comment

Uh oh!

LoekL commented Jun 24, 2020

Uh oh!

slevenick commented Jun 24, 2020

Uh oh!

ffung commented Apr 3, 2020 •

edited by slevenick

Loading