/
channels.ad
97 lines (71 loc) · 6.43 KB
/
channels.ad
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
=== Channels
Once group links and routers are defined, configuration must be completed to specify which data (tables, file systems, etc.) should be
synchronized over those links and routers. The next step in defining which specific data in the database is moved is to define logical
groupings for that data. Channels define those logical groupings. As an example, a set of tables that hold customer data might be
logically grouped together in a Customer channel. Sales, returns, tenders, etc. (transaction data) might be logically grouped into a
transaction channel. A default channel is automatically created that all tables will fall into unless other channels are created and
specified. The default channel is called 'default'.
Channels can be disabled, suspended, or scheduled as needed.
ifdef::pro[]
image::channel.png[]
endif::pro[]
WARNING: Transactions will NOT be preserved across channels so its important to setup channels to contain all tables that participate in a given transaction.
ifdef::pro[]
.Required Fields
endif::pro[]
Channel ID:: Identifier used through the system to identify a given channel.
[[processing-order]]Processing Order:: Numeric value to determine the order in which a channel will be processed. Channels will be processed in ascending order.
[[batch-algorithm]]Batch Algorithm:: Batching is the grouping of data, by channel, to be transferred and committed at the client together.
.Channel Batching Algorithms
|===
|Default|All changes that happen in a transaction are guaranteed to be batched together. Multiple transactions will be batched and committed together until there is no more data to be sent or the max_batch_size is reached.
|Transactional|Batches will map directly to database transactions. If there are many small database transactions, then there will be many batches. The max_batch_size column has no effect.
|Nontransactional|Multiple transactions will be batched and committed together until there is no more data to be sent or the max_batch_size is reached. The batch will be cut off at the max_batch_size regardless of whether it is in the middle of a transaction.
|===
[[max-batch-size]]Max Batch Size:: Specifies the maximum number of data events to process within a batch for this channel.
[[max-batch-to-send]]Max Batch To Send:: Specifies the maximum number of batches to send for a given channel during a 'synchronization' between two nodes. A 'synchronization' is equivalent to a push or a pull. For example, if there are 12 batches ready to be sent for a channel and max_batch_to_send is equal to 10, then only the first 10 batches will be sent even though 12 batches are ready.
[[max-data-to-route]]Max Data To Route:: Specifies the maximum number of data rows to route for a channel at a time.
[[max-network-kbps]]Max KB/s:: Specifies the maximum network transfer rate in kilobytes per second. Use zero to indicate unlimited. When throttling the channel, make sure the channel is on its own queue or within a queue of channels that are throttled at the same rate. This is currently only implemented when staging is enabled.
Data Loader Types:: Determines how data will be loaded into the target tables. These are used during an initial load or a reverse initial load. Data loaders do not always
have to load into the target relational database. They can write to a file, a web service, or any other type of non-relational data source.
Data loaders can also use other techniques to increase performance of data loads into the target relation database.
|===
|default|Performs an insert first and if this fails will fall back to an update to load the data.
|ftp_localhost|Sends the data in CSV format to a configured ftp location. These locations are setup in the TODO {SYM_HOME}/conf/ftp-extensions.xml
|bulk|Assigns the appropriate bulk loader to this channel. Supported bulk loaders include: Microsoft SQL, PostgreSQL, MySQL and Amazon Redshift over S3.
|mongodb|MongoDB data loader.
|===
TIP: Tables that should be data loaded should be configured to use this channel. Many times, a reload channel will
be set to bulk load to increase the performance of an initial load.
ifdef::pro[]
.Advanced Options
endif::pro[]
Group Link Direction:: For a node group link that is reversible, the channel can specify either "push" or "pull" to override the default group link communication. If this field is empty, the default group link communication is used.
Enabled:: Indicates whether the channel is enabled or disabled. If a channel is disabled, data is still captured for changes
that occur on the source system, but it will not be routed and sent to the target until the channel is re-enabled.
Reload Channel:: Indicates whether a channel is available for initial loads and reverse initial loads.
File Sync Channel:: Indicates whether a channel is available for file synchronization.
Use Old Data To Route:: Indicates if the old data will be included for routing. Routing can then use this data for processing. Defaults to true.
Use Row Data To Route:: Indicates if the current data will be included for routing. Routing can then use this data for processing. Defaults to true.
Use Primary Key (PK) Data to Route:: Indicates if the primary key data will be include for routing. For example maybe a store ID is needed to apply logic on before sending to the appropriate target nodes. Defaults to true.
Tables Contain Big Lobs:: Indicates whether the channel contains big lobs. Some databases have shortcuts that SymmetricDS can take advantage of if it knows that the lob columns in SYM_DATA aren't going to contain large lobs. The definition of how large a 'big' lob is varies from database to database.
.Sample Channels
=====
ifndef::pro[]
[source,sql]
----
insert into SYM_CHANNEL (channel_id, processing_order, max_batch_size, max_batch_to_send,
extract_period_millis, batch_algorithm, enabled, description)
values ('item', 10, 1000, 10, 0, 'default', 1, 'Item and pricing data');
insert into SYM_CHANNEL (channel_id, processing_order, max_batch_size,
max_batch_to_send, extract_period_millis, batch_algorithm, enabled, description)
values ('sale_transaction', 1, 1000, 10, 60000,
'transactional', 1, 'retail sale transactions from register');
----
endif::pro[]
ifdef::pro[]
TODO
endif::pro[]
=====
.Channel Tips and Tricks
TIP: Increase performance by creating designated channels for tables that use LOB data types. For these channels be sure to check the "Table Contains Big Lobs" to increase performance.