/
bigquery.ad
75 lines (45 loc) · 2.15 KB
/
bigquery.ad
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
=== Google Big Query
Send changes from your relational database to Google's Big Query.
==== Setup
Big Query is only supported as a load only node in SymmetricDS. See <<Load Only Node >> for details on setting up a load only node in SymmetricDS.
ifdef::pro[]
Setup the Big Query node by using the <<Add Node>> wizard and selecting Big Query as the type.
image::appendix/bigquery-database-settings.png[]
After hitting next you can setup advanced options for your Snowflake node.
endif::pro[]
ifndef::pro[]
.Example properties to setup a Google Big Query load only node
----
load.only=true
target.db.url=bigquery\://cloud.google.com
target.db.driver=google
----
endif::pro[]
==== Loading Data Into Big Query
ifndef::pro[]
===== Setup reload channels for bulk loading.
Update any reload channels that will be used on the table triggers that will capture changes and send them to Big Query by setting the column data_loader_type to 'bulk'. It is also recommended to increase the batch size so that larger CSV files will be processed instead of the default size on reloads of 10,000 rows.
.Example SQL to setup the main reload channel to use bulk and also update the batch sizes.
[source, SQL]
----
update sym_channel set data_loader_type='bulk', max_batch_size=500000 where channel_id='reload'
----
endif::pro[]
===== Big Query Authentication
Create a JSON credentials file through your Big Query account
https://cloud.google.com/docs/authentication/getting-started
ifdef::pro[]
Provide this file path on the advanced settings while setting up a Big Query node. The advanced settings also requires that you provide a project ID and location for your Big Query project.
====
NOTE: You will need to use your Google Big Query dataset name in the target schema of the router that is used to route data to Big Query.
====
image::appendix/bigquery-advanced-settings-snowflake-managed.png[]
endif::pro[]
ifndef::pro[]
[source, properties]
----
google.bigquery.project.id=<Google Big Query Project ID>
google.bigquery.security.credentials.path=<Local Path to JSON Credentials File>
google.bigquery.location=<Google Big Query Location: defaults to US>
----
endif::pro[]