-
Notifications
You must be signed in to change notification settings - Fork 44
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
gcp/bigquery support #279
Comments
note: https://stackoverflow.com/questions/50470044/syntax-error-expected-in-bigquery -- no varchar, no bigint, no real. easy to search-replace in features-v2.sql |
https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-json -- this seems like the best way to import data, quite similar to how kinesis is writing to s3 and that gets loaded into redshift |
so the UI for that is a bit weird... one can load into existing tables by creating a new table. |
ah no... extract.js converts booleans:
I wonder if that is legacy or actually required for redshift. If it is then this should be moved to database.js |
ref #279 do not convert booleans to 0/1. This was done for redshift but breaks bigquery when trying to load files.
ref #279 do not convert booleans to 0/1. This was done for redshift but breaks bigquery when trying to load files.
I think the best way to load it is to buffer 100 entries (or 60 seconds) or so as a line-json file and then use load() https://cloud.google.com/bigquery/docs/loading-data-local?hl=en#loading_data_from_a_local_data_source |
https://cloud.google.com/bigquery/streaming-data-into-bigquery#bigquery_table_insert_rows-nodejs
https://cloud.google.com/bigquery-transfer/docs/redshift-migration (the features-v2 schema should just work)
cc @juandebravo @jbgwsr
The text was updated successfully, but these errors were encountered: