google bigquery implementation complete

[mohanish2504]

/claim #115
/closes #115

• QueryCSV and QueryJSON
• InsertFromNDJson: Uploads to CSV to GCS and then to Bigquery
• Added GCS storage provider (gcs and s3 are interoperable, tried with but didn't work)
• Optional GCS delete
• Optional GCS prefix

@poundifdef Some thigs to take care:

• Both query fromats require's user to query with dataset.table, may be need to somewhere in docs

Screenshots:

[mohanish2504]

@poundifdef this is ready for review

[poundifdef]

Thanks for the quick work here. I've left some comments and questions.

[mohanish2504]

@poundifdef this is ready for review

[poundifdef]

Thank you - I've tested this and it is almost ready to go. Just a few small changes.

[mohanish2504]

@poundifdef added changes

[poundifdef]

This branch does not work any longer.

curl -X POST 'http://localhost:8080/api/data/insert/my_dataset.t?api_key=bq' --data '{"x": "y"}'

Gives this error:

14:41:55 ERR insert.go:23 > CreateEmptyTable: failed to create Table error="googleapi: Error 404: Not found: Dataset scratch-data-410814:my_dataset was not found in location US, notFound" query="CREATE TABLE IF NOT EXISTS my_dataset.t (__row_id BIGINT)"
14:41:55 ERR workers.go:41 > Unable to process message error="googleapi: Error 404: Not found: Dataset scratch-data-410814:my_dataset was not found in location US, notFound" thread=0

You probably want to use this api, right? https://cloud.google.com/bigquery/docs/datasets#go

[mohanish2504]

So split and create the dataset first?

[poundifdef]

So split and create the dataset first?

Yes. The goal is for the user to do as little configuration as possible, so if we can automate creating a dataset then that is what I'd like to do.

[mohanish2504]

I will need location parameter, take it from user in config?

[poundifdef]

I will need location parameter, take it from user in config?

Yes

[mohanish2504]

How about get query here? What if dataset and table not there? Create and return empty?
Most likely user won't be querying such table.

[poundifdef]

How about get query here? What if dataset and table not there? Create and return empty? Most likely user won't be querying such table.

If a user tries to query and the dataset or table does not exist, then we should just return the error that BigQuery throws. We only automatically create tables when the user inserts data.

[mohanish2504]

How about get query here? What if dataset and table not there? Create and return empty? Most likely user won't be querying such table.

If a user tries to query and the dataset or table does not exist, then we should just return the error that BigQuery throws. We only automatically create tables when the user inserts data.

That is exactly for current case, pushing.

[mohanish2504]

@poundifdef pushed the changes

[poundifdef]

Thank you for your persistence. I still get an error when running the same test as above:

curl -X POST 'http://localhost:8080/api/data/insert/my_dataset.t?api_key=bq' --data '{"x": "y"}'

Here is the error:

17:02:36 TRC types.go:59 > column_type_counts={"__row_id":{"int":1},"x":{"string":1}}
17:02:36 TRC types.go:86 > column_types={"__row_id":"int","x":"string"}
17:02:37 INF insert.go:119 > Uploading file to GCS 
17:02:38 INF insert.go:125 > Uploaded file to GCS gcs_file=b/my_dataset.t2/1_my_dataset.t2_1769831789700034560.ndjson.ndjson
17:02:38 INF insert.go:127 > Streaming data to BigQuery
17:02:39 ERR insert.go:172 > StreamDataToBigQuery: failed to stream data to BigQuery error="googleapi: Error 400: Invalid schema update. Field x has changed type from STRING to BOOLEAN, invalid" query="LOAD DATA INTO my_dataset.t2 FROM FILES ( format = 'JSON', uris = ['gs://scratchdata-test-bucket/b/my_dataset.t2/1_my_dataset.t2_1769831789700034560.ndjson.ndjson'] )"
17:02:39 ERR insert.go:130 > Failed to stream data to BigQuery error="googleapi: Error 400: Invalid schema update. Field x has changed type from STRING to BOOLEAN, invalid"
17:02:39 ERR insert.go:184 > Failed to upload and stream data to BigQuery error="googleapi: Error 400: Invalid schema update. Field x has changed type from STRING to BOOLEAN, invalid" file=data/worker/1_my_dataset.t2_1769831789700034560.ndjson.ndjson table=my_dataset.t2
17:02:39 ERR workers.go:41 > Unable to process message error="googleapi: Error 400: Invalid schema update. Field x has changed type from STRING to BOOLEAN, invalid" thread=0

I believe this is because BigQuery's autodetect is converting the string "y" into a boolean before inserting.

[mohanish2504]

You are right @poundifdef, that's the case of autodetection
Thanks for being pateint.

[mohanish2504]

@poundifdef got chance to test this?