Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
b157b0c
commit f1fa5f2
Showing
5 changed files
with
88 additions
and
2 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,75 @@ | ||
=== Google Big Query | ||
|
||
Send changes from your relational database to Google's Big Query. | ||
|
||
==== Setup | ||
|
||
Big Query is only supported as a load only node in SymmetricDS. See <<Load Only Node >> for details on setting up a load only node in SymmetricDS. | ||
|
||
ifdef::pro[] | ||
Setup the Big Query node by using the <<Add Node>> wizard and selecting Big Query as the type. | ||
|
||
image::appendix/bigquery-database-settings.png[] | ||
|
||
After hitting next you can setup advanced options for your Snowflake node. | ||
|
||
endif::pro[] | ||
|
||
ifndef::pro[] | ||
|
||
.Example properties to setup a Google Big Query load only node | ||
---- | ||
load.only=true | ||
|
||
target.db.url=bigquery\://cloud.google.com | ||
target.db.driver=google | ||
|
||
|
||
---- | ||
|
||
endif::pro[] | ||
|
||
==== Loading Data Into Big Query | ||
|
||
|
||
|
||
ifndef::pro[] | ||
===== Setup reload channels for bulk loading. | ||
|
||
Update any reload channels that will be used on the table triggers that will capture changes and send them to Big Query by setting the column data_loader_type to 'bulk'. It is also recommended to increase the batch size so that larger CSV files will be processed instead of the default size on reloads of 10,000 rows. | ||
|
||
|
||
.Example SQL to setup the main reload channel to use bulk and also update the batch sizes. | ||
[source, SQL] | ||
---- | ||
update sym_channel set data_loader_type='bulk', max_batch_size=500000 where channel_id='reload' | ||
---- | ||
endif::pro[] | ||
|
||
===== Big Query Authentication | ||
|
||
Create a JSON credentials file through your Big Query account | ||
|
||
https://cloud.google.com/docs/authentication/getting-started | ||
|
||
|
||
ifdef::pro[] | ||
|
||
Provide this file path on the advanced settings while setting up a Big Query node. The advanced settings also requires that you provide a project ID and location for your Big Query project. | ||
|
||
==== | ||
NOTE: You will need to use your Google Big Query dataset name in the target schema of the router that is used to route data to Big Query. | ||
==== | ||
|
||
image::appendix/bigquery-advanced-settings-snowflake-managed.png[] | ||
endif::pro[] | ||
ifndef::pro[] | ||
[source, properties] | ||
---- | ||
google.bigquery.project.id=<Google Big Query Project ID> | ||
google.bigquery.security.credentials.path=<Local Path to JSON Credentials File> | ||
google.bigquery.location=<Google Big Query Location: defaults to US> | ||
|
||
---- | ||
endif::pro[] | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file added
BIN
+41 KB
symmetric-assemble/src/asciidoc/images/appendix/bigquery-advanced-settings.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+48.6 KB
symmetric-assemble/src/asciidoc/images/appendix/bigquery-database-settings.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.