Google Cloud: BigQuery DataSet and Table resources #3764

coffeepac · 2015-11-05T05:10:08Z

BigQuery is a managed database-like service. It supports SQL as
a query language and nearly infinite scale. This PR adds support
to manage tables and datasets.

coffeepac · 2015-11-20T05:07:20Z

@lwander @sparkprime anything I can do for this one or is it in the hopper, waiting its turn?

lwander · 2015-11-20T13:55:10Z

builtin/providers/google/resource_bigquery_dataset.go

+
+		Schema: map[string]*schema.Schema{
+			"name": &schema.Schema{
+				Type:     schema.TypeString,


It looks like you're missing quite a few features described here

I am missing two general classes of things:

the role access information, which should be included. I'll do that

updating/patching an existing dataset. Update doesn't need to be implemented at all, in terraform's context that's handled by delete/create when a ForceNow config val is set. I will use patch to implement Update

Anything else you'd like from that page? I see list, which isn't useful in a terraform context, and a bunch of computed fields that don't have any obvious use in a terraform context.

For starters

datasetReference is described as being required. This resource as is shouldn't work as is.

Otherwise,

access[]

friendlyName (which you've called name)

description

defaultTableExpirationMs

location
are optional, but nice to have.

When it comes to exported attributes

self_link is important to include as computed, since other resources often refer to one another by GCE assigned URI.

id for similar reasons.

lastModifiedTime can be useful for people examining infrastructure managed by Terraform.

coffeepac · 2015-11-20T20:08:09Z

@lwander Thanks man! I really appreciate you taking the time to review!

lwander · 2015-11-20T20:21:10Z

@coffeepac you got it! Thanks for doing all this work so far.

coffeepac · 2015-12-04T07:21:04Z

@lwander I wanted to get your thoughts on my changes to the Dataset resource before I do something similar to the table and then go implement the Patch operation.

The only comment I didn't directly implement was the comment about 'datasetReference'. This is a struct composed of datasetId and projectId which I'm pulling from the provider configuration. If that's wrong/inconsistent lemme know and I'll rework it.

lwander · 2015-12-04T14:22:54Z

builtin/providers/google/resource_bigquery_dataset.go

+				Optional: true,
+			},
+
+			"access": &schema.Schema{


This gets a bit tricky, since we implemented ACLs for GCS a little differently, but that was only possible because the API is different. The idea there was, you just provide a list of role entity pairs like this. The GCS API basically squashed userByEmail, groupByEmail, and domain all into one field, so all we had to do was have the user concatenate the contents of that field with the role, and that would be the access control object. Second, there was no view type object attached with each GCS ACL, since the ACL is an entirely separate resource for GCS objects and buckets. Basically, to match the GCS Terraform interface, we'd really have to contort the BigQuery API, which is a shame.

I'd say what you have here is best, I'm just leaving this comment in case people have questions about this decision in the future.

@sparkprime, what do you think?

lwander · 2015-12-04T14:33:26Z

When it comes to the datasetReference, you're totally right for dropping projectId, so what you have looks good.

Are you planning on flushing out the table resource as well? It has quite a few fields here.

coffeepac · 2015-12-04T14:48:43Z

I am planning on fleshing out the table resource as well. I wanted some
feedback before I did it though. Make sure I'm on the right path. Thanks
for the feedback!
On Dec 4, 2015 6:33 AM, "Lars Wander" notifications@github.com wrote:

When it comes to the datasetReference, you're totally right for dropping
projectId, so what you have looks good.

Are you planning on flushing out the table resource as well? It has quite
a few fields here
https://cloud.google.com/bigquery/docs/reference/v2/tables.

—
Reply to this email directly or view it on GitHub
#3764 (comment).

BigQuery is a managed database-like service. It supports SQL as a query language and nearly infinite scale. This PR adds support to manage tables and datasets. first cut, need to finish tests fully functional and the tests are looking good. this is a really thin wrapper around the create/delete portion of the API removed optional parameters, there is no update defined add docs for bigquery resources

I'm going to remove all but the top level of fields and add a json file config option as well. but I wanted to save this incase I want it arbitarily in the future

the table schema is an arbitrarily deep structure which I'm not sure how to represent in terraform's schema syntax. failing that I have introduced a work around: - one level deep of table structure in the standard terraform schema manner (as many fields as you want, no sub fields or fields of type 'record') - arbitrary depth but specify a file that has the layout in json I really like the idea of including an auxilarly file to represent the table structure but I'm open to it being spcified via the file built-in or something. I don't really want to let there be arbitrary depth configs in the terraform config file. that seems a bit heinous

saving because HDD is wonky. will squash later

now to add some tests for the new schema nonsense

adds a test for the inline table format (one level) and the json file format (arbitrary depth)

coffeepac · 2016-01-06T06:43:46Z

@lwander oh holidays... I haven't started working on patch semantics yet but I have added bigquery table. tables can have an arbitrary depth of record fields and I didn't see a clean way of doing that with helper/schema so I fudged it by having the schema in code allow for depth of one but I added a schemaFile that expects json and allows for arbitrary depth.

the code to support arbitrary depth of record fields is there so if there's a clean way to add them into the schema I'm all for it if you have a way.

either way, I like having the schema in a separate file so if there is a 4 layer deep table (my current use case) it isn't garbaging up the tf file. sound okay or is that no go?

lwander · 2016-01-13T22:26:51Z

Could you point me to the schemaFile attribute? Since Terraform doesn't have an expressive enough syntax schemaFile sounds like a much better solution.

coffeepac · 2016-01-13T22:49:47Z

line 46 in builtin/providers/google/resource_bigquery_table.go

test using this file is at line 29 and the resource is defined at 105 in builtin/providers/google/resource_bigquery_table_test.go

the test fixture for this is: builtin/providers/google/test-fixtures/fake_bigquery_table.json

the test schemaFile goes three levels deep.

heimweh · 2017-03-01T10:20:30Z

@coffeepac I'd love to see support for BigQuery in Terraform.
I can see that this PR hasn't been updated in a while,
do you plan to continue your work on this or do you mind if I take a stab at this? :)

coffeepac · 2017-03-01T17:46:08Z

@heimweh feel free to take it over! I got pulled off and had to create an external plugin for this. its here: https://github.com/22Acacia/terraform-provider-googlebigquery Its currently in production and does what we need so we're just living with it for now.

The big chunk of work I never got to was how to implement Update, its currently a big ball of lying No-Op. The golang docs for the bigquery library have taken a massive leap forward in the past year so it shouldn't be a slog anymore. It was basically just the auto-generated go doc stuff.

Lemme know if you want me to review anything, open up permissions on a branch somewhere or have more questions. I'm happy to help!

kurtmaile · 2017-07-02T10:21:13Z

Hi all, Any plans to provide a resource for bigquery views? Would be super helpful!!

Thanks for your hard work on these core resources.....

kurtmaile · 2017-07-02T10:22:02Z

Hi all, Any plans to provide a resource for bigquery views? Would be super helpful!!

Thanks for your hard work on these core resources.....

coffeepac · 2017-09-07T20:05:42Z

this was merged elsewhere. going to close this PR.

ghost · 2020-04-07T02:14:06Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

catsby added enhancement provider/google-cloud labels Nov 5, 2015

coffeepac changed the title ~~Clean commit of adding bigquery resources~~ Google Cloud: BigQuery DataSet and Table resources Nov 6, 2015

coffeepac force-pushed the google-bigquery branch from bfeaa9b to eddbcd1 Compare November 20, 2015 07:38

lwander reviewed Nov 20, 2015
View reviewed changes

coffeepac force-pushed the google-bigquery branch from eddbcd1 to d1e7aa6 Compare December 4, 2015 07:18

lwander reviewed Dec 4, 2015
View reviewed changes

coffeepac added 6 commits January 4, 2016 20:54

add flag so deletes only remove bq artifacts if set

f440a28

add can_delete optional param

191b7fc

answer comments from @lwander

00f0c1b

add flag so deletes only remove bq artifacts if set

e6a59d8

answer comments from @lwander

9132344

coffeepac force-pushed the google-bigquery branch from 98c2579 to 9132344 Compare January 5, 2016 04:57

coffeepac added 5 commits January 4, 2016 22:06

remove can_delete, add insanity layers of fields to TableSchema

a31f5a5

I'm going to remove all but the top level of fields and add a json file config option as well. but I wanted to save this incase I want it arbitarily in the future

code compiles, not much else

e9b39d1

saving because HDD is wonky. will squash later

code complete, original tests pass

27b8418

now to add some tests for the new schema nonsense

add tests for field format

7850f11

adds a test for the inline table format (one level) and the json file format (arbitrary depth)

heimweh mentioned this pull request Apr 6, 2017

provider/google: BigQuery Dataset #13436

Merged

coffeepac closed this Sep 7, 2017

ghost locked and limited conversation to collaborators Apr 7, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Google Cloud: BigQuery DataSet and Table resources #3764

Google Cloud: BigQuery DataSet and Table resources #3764

coffeepac commented Nov 5, 2015

coffeepac commented Nov 20, 2015

lwander Nov 20, 2015

coffeepac Nov 20, 2015

lwander Nov 20, 2015

coffeepac commented Nov 20, 2015

lwander commented Nov 20, 2015

coffeepac commented Dec 4, 2015

lwander Dec 4, 2015

lwander commented Dec 4, 2015

coffeepac commented Dec 4, 2015

coffeepac commented Jan 6, 2016

lwander commented Jan 13, 2016

coffeepac commented Jan 13, 2016

heimweh commented Mar 1, 2017

coffeepac commented Mar 1, 2017

kurtmaile commented Jul 2, 2017

kurtmaile commented Jul 2, 2017

coffeepac commented Sep 7, 2017

ghost commented Apr 7, 2020

Google Cloud: BigQuery DataSet and Table resources #3764

Google Cloud: BigQuery DataSet and Table resources #3764

Conversation

coffeepac commented Nov 5, 2015

coffeepac commented Nov 20, 2015

lwander Nov 20, 2015

Choose a reason for hiding this comment

coffeepac Nov 20, 2015

Choose a reason for hiding this comment

lwander Nov 20, 2015

Choose a reason for hiding this comment

coffeepac commented Nov 20, 2015

lwander commented Nov 20, 2015

coffeepac commented Dec 4, 2015

lwander Dec 4, 2015

Choose a reason for hiding this comment

lwander commented Dec 4, 2015

coffeepac commented Dec 4, 2015

coffeepac commented Jan 6, 2016

lwander commented Jan 13, 2016

coffeepac commented Jan 13, 2016

heimweh commented Mar 1, 2017

coffeepac commented Mar 1, 2017

kurtmaile commented Jul 2, 2017

kurtmaile commented Jul 2, 2017

coffeepac commented Sep 7, 2017

ghost commented Apr 7, 2020