Skip to content

Commit

Permalink
0004190: Google Big Query Support
Browse files Browse the repository at this point in the history
  • Loading branch information
jumpmind-josh committed Dec 10, 2019
1 parent b157b0c commit f1fa5f2
Show file tree
Hide file tree
Showing 5 changed files with 88 additions and 2 deletions.
75 changes: 75 additions & 0 deletions symmetric-assemble/src/asciidoc/appendix/bigquery.ad
@@ -0,0 +1,75 @@
=== Google Big Query

Send changes from your relational database to Google's Big Query.

==== Setup

Big Query is only supported as a load only node in SymmetricDS. See <<Load Only Node >> for details on setting up a load only node in SymmetricDS.

ifdef::pro[]
Setup the Big Query node by using the <<Add Node>> wizard and selecting Big Query as the type.

image::appendix/bigquery-database-settings.png[]

After hitting next you can setup advanced options for your Snowflake node.

endif::pro[]

ifndef::pro[]

.Example properties to setup a Google Big Query load only node
----
load.only=true

target.db.url=bigquery\://cloud.google.com
target.db.driver=google


----

endif::pro[]

==== Loading Data Into Big Query



ifndef::pro[]
===== Setup reload channels for bulk loading.

Update any reload channels that will be used on the table triggers that will capture changes and send them to Big Query by setting the column data_loader_type to 'bulk'. It is also recommended to increase the batch size so that larger CSV files will be processed instead of the default size on reloads of 10,000 rows.


.Example SQL to setup the main reload channel to use bulk and also update the batch sizes.
[source, SQL]
----
update sym_channel set data_loader_type='bulk', max_batch_size=500000 where channel_id='reload'
----
endif::pro[]

===== Big Query Authentication

Create a JSON credentials file through your Big Query account

https://cloud.google.com/docs/authentication/getting-started


ifdef::pro[]

Provide this file path on the advanced settings while setting up a Big Query node. The advanced settings also requires that you provide a project ID and location for your Big Query project.

====
NOTE: You will need to use your Google Big Query dataset name in the target schema of the router that is used to route data to Big Query.
====

image::appendix/bigquery-advanced-settings-snowflake-managed.png[]
endif::pro[]
ifndef::pro[]
[source, properties]
----
google.bigquery.project.id=<Google Big Query Project ID>
google.bigquery.security.credentials.path=<Local Path to JSON Credentials File>
google.bigquery.location=<Google Big Query Location: defaults to US>

----
endif::pro[]

13 changes: 12 additions & 1 deletion symmetric-assemble/src/asciidoc/appendix/databases.ad
Expand Up @@ -22,6 +22,17 @@ by database.
|Transactional DDL
|Load Only

|Big Query
|All
|N
|N
|N
|Y
|N
|N
|N
|Y

|DB2
|9.5
|N
Expand Down Expand Up @@ -486,7 +497,7 @@ When locating a table, SymmetricDS uses the default catalog and schema unless th
|select sys_context('USERENV', 'CURRENT_SCHEMA') from dual

|===

include::bigquery.ad[]
include::db2.ad[]
include::derby.ad[]
include::firebird.ad[]
Expand Down
2 changes: 1 addition & 1 deletion symmetric-assemble/src/asciidoc/appendix/snowflake.ad
Expand Up @@ -4,7 +4,7 @@ Send changes from your relational database to Snowflake.

==== Setup

Snowflake is only support as a load only node in SymmetricDS. See <<Load Only Node >> for details on setting up a load only node in SymmetricDS.
Snowflake is only supported as a load only node in SymmetricDS. See <<Load Only Node >> for details on setting up a load only node in SymmetricDS.


ifdef::pro[]
Expand Down
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit f1fa5f2

Please sign in to comment.