/
manage.ad
277 lines (183 loc) · 10.8 KB
/
manage.ad
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
== Manage
This section will talk about how to manage and monitor SymmetricDS.
=== Nodes
ifdef::pro[]
Nodes can be added and managed in the Nodes screen under the Manage tab.
image::images/manage/manage-nodes.png[]
endif::pro[]
==== Add Node
Multiple nodes can be hosted in a single SymmetricDS instance. SymmetricDS will start a node for each properties file it finds in the engines directory.
ifndef::pro[]
Additional nodes can be added to the same SymmetricDS instance that the master node is running in or they can be added to a different
SymmetricDS instance. Either way, you create additional nodes by creating an <<Node Properties File>> with the `registration.url` set to
the `sync.url` of the master node. When the SymmetricDS instance is restarted the new node will attempt to register with the master node.
For the new node to fully become a part of the synchronization network <<Registration>> must be opened.
endif::pro[]
ifdef::pro[]
Add nodes to your synchronization scenario by using the Connect Database Wizard. This wizard is automatically shown when less the 2 nodes are configured.
It can also be accessed from the Manage Nodes screen by clicking the *Add* button.
image::images/wizard-connect/wizard-connect-menu.png[]
include::manage/node-add.ad[]
endif::pro[]
==== Load Data
include::manage/node-load.ad[]
==== Control
include::manage/node-control.ad[]
==== Registration
include::manage/node-registration.ad[]
==== Push Registration
include::manage/node-push-registration.ad[]
==== Initial Loads
Loading data for 3.8 and above has been modified, see <<Load Data>>.
ifndef::pro[]
When a load is requested it will either set the `initial_load_enabled` or the `reverse_initial_load_enabled` flag on the
appropriate <<NODE_SECURITY>> row.
When the <<Route Job>> runs next, it will create batches that represent the initial load. Batches will be created on the reload
channel for each table that is defined by <<Table Triggers>> and linked by <<Table Routing>> in the direction that the load
was requested. The default reload channel is the _"reload"_ channel. At the
same time reload batches are inserted, all previously pending batches
for the node are marked as successfully sent.
Each table defined by <<Table Triggers>> and linked by <<Table Routing>> is represented by a reload <<OUTGOING_BATCH>>. The batches
are inserted in the defined order. If the `initial_load_order` is the same then SymmetricDS tries to determine the order the
tables need to be loaded in automatically based on foreign key dependencies. A negative value for `initial_load_order` in <<Table Routing>> will result
_no_ reload batch being inserted.
If there are cyclical constraints, then foreign keys might need to be
turned off or the initial load will need to be manually configured based
on knowledge of how the data is structured.
A SQL statement is run against each table to get the data load
that will be streamed to the target node. The selected data is filtered
through the configured router for the table being loaded. If the data
set is going to be large, then SQL criteria can optionally be provided in `initial_load_select`
to pare down the data that is selected out of the database.
IMPORTANT: Note that if the parent node that a node is registering with is _not_ a registration server node
(as can happen when using <<REGISTRATION_REDIRECT>> or when using multiple tiers) the parent node's <<NODE_SECURITY>>
entry must exist at the parent node and have a non-null value for column `initial_load_time`. Nodes can't be registered
to a non-registration-server node without this value being set one way or another (i.e., manually, or as a result
of an initial load occurring at the parent node).
===== Partial Initial Loads
An efficient way to select a subset of data from a table for an initial
load is to provide an `initial_load_select` clause in <<Table Routing>>
. This clause, if present, is applied as a `where` clause to the SQL used to
select the data to be loaded. The clause may use "t" as an alias for the table being loaded, if needed. The
`$(externalId)` token can be used for subsetting the data in the where clause.
In cases where routing is done using a feature like
<<Subselect Router>>
, an
`initial_load_select`
clause matching the subselect's criteria would be a more efficient
approach. Some routers will check to see if the
`initial_load_select`
clause is provided, and they will
_not_
execute assuming that the more optimal path is using the
`initial_load_select`
statement.
One example of the use of an initial load select would be if you wished
to only load data created more recently than the start of year 2011.
Say, for example, the column
`created_time`
contains the creation date. Your
`initial_load_select`
would read
`created_time > ts {'2011-01-01 00:00:00.0000'}`
(using whatever timestamp format works for your database). This then
gets applied as a
`where`
clause when selecting data from the table.
IMPORTANT: When providing an
`initial_load_select`
be sure to test out the criteria against production data in a query
browser. Do an explain plan to make sure you are properly using indexes.
endif::pro[]
include::manage/node-initial-load.ad[]
==== Send
include::manage/node-send.ad[]
=== Jobs
Most work done by SymmetricDS is initiated by jobs. Jobs are tasks that are started and scheduled by a job manager.
Jobs are enabled by the `start.<name>.job` parameter.
Most jobs are enabled by default. The frequency at which a job
runs is controlled by one of two parameters:
* `job.<name>.period.time.ms`
* `job.<name>.cron`
If a valid cron property exists in the configuration, then it will be used to schedule the job. Otherwise, the job manager
will attempt to use the period.time.ms property.
ifdef::pro[]
From the manage jobs screen the frequency that a job will run for the *current node* in the cluster can be managed. If you want to change
the frequency a job runs across a group or for everyone, you modify the parameter in the <<Parameters>> screen.
A job can also be _stopped_ and _restarted_. If a job is scheduled to run infrequently and the job is needed to run immediately
you can select job and click *Run Now*.
image::images/manage/manage-jobs.png[]
endif::pro[]
ifndef::pro[]
The frequency of jobs can be configured in either the <<Node Properties File>> or in the <<PARAMETER>> table. When managed in
<<PARAMETER>> table the frequency properties can be changed in the master node and when the updated settings sync to the other nodes
in the system the job manager will restart the jobs at the new frequency settings.
endif::pro[]
SymmetricDS utilizes Spring's CRON support, which includes seconds as the first parameter. This differs from the typical Unix-based
implementation, where the first parameter is usually minutes. For example,
`*/15 * * * * *`
means every 15 seconds, not every 15 minutes. See
http://static.springsource.org/spring/docs/3.0.x/javadoc-api/org/springframework/scheduling/support/CronSequenceGenerator.html"[Spring's
documentation]
for more details.
include::manage/jobs.ad[]
=== Installed Triggers
include::manage/installed-triggers.ad[]
ifdef::pro[]
=== Outgoing Loads
The Outgoing Loads screen shows the number of outgoing loads that have been queued. It also lists loads that have had <<Outgoing Batches>> created. The screen shows
loads that are sourced from the current node.
You can cancel a load that is in progress by selecting the load and pressing the _Cancel_ button.
image::images/manage/manage-outgoing-loads.png[]
=== Incoming Loads
The Incoming Loads screen shows the number of incoming loads that have been queued. It also lists loads that have had <<Incoming Batches>> created. The screen shows
loads that are targeted at the current node.
You can cancel a load that is in progress by selecting the load and pressing the _Cancel_ button.
image::images/manage/manage-incoming-loads.png[]
endif::pro[]
=== Outgoing Batches
ifdef::pro[]
The Outgoing Batches screens allows you to monitor data that has been <<_route_job,routed>>.
image::images/manage/manage-outgoing-batches.png[]
endif::pro[]
Outgoing batches are delivered to the target node when the source node <<_push_job,pushes>> or when the target node <<_pull_job,pulls>>.
A single push or pull connection is called a _synchronization_.
For one synchronization, each enabled channel will be processed. Channels are processed in the order defined by the
<<configuration/channels.ad#processing-order,Processing Order>>
setting on the <<_channels,channel>> with two exceptions:
* If there are reload channels available to be sent _and_ the reload channels are _not_ in error, then only reload channels will be sent
* If a channel is in error it will be moved to the bottom of the list
When outgoing batches are selected for a node and a channel, the maximum number of batches that are extracted per synchronization is
controlled by the <<configuration/channels.ad#max-batch-to-send,Max Batch To Send>> setting on the <<_channels,channel>>.
There is also a setting that controls the max number of bytes to send in one synchronization. If SymmetricDS has extracted more than
the number of bytes configured by the `transport.max.bytes.to.sync` parameter, then it will finish extracting the current batch and then finish
synchronization so the client has a chance to process and acknowledge the "big" batch. This may happen before the configured
<<configuration/channels.ad#max-batch-to-send,Max Batch To Send>> has been reached.
When extracting a batch, data is first extracted to the <<Staging Area>> and then sent across the network from the <<Staging Area>>.
The staging area is used to minimize the amount of time a database connection is being used when streaming
over slower networks. The use of the staging area can be turned off by setting the `stream.to.file.enabled` parameter.
include::manage/outgoing-batches.ad[]
=== Incoming Batches
ifdef::pro[]
The Incoming Batches screens allows you to monitor batches that have been delivered to the target node.
image::images/manage/manage-incoming-batches.png[]
endif::pro[]
Incoming batches are delivered to the target node when the source node <<_push_job,pushes>> or when the target node <<_pull_job,pulls>>.
Incoming batches are written to the <<Staging Area>> first and then loaded. The use of the staging area can be turned off by setting the `stream.to.file.enabled` parameter.
include::manage/incoming-batches.ad[]
include::manage/staging.ad[]
ifdef::pro[]
include::manage/processes.ad[]
endif::pro[]
include::manage/pull-threads.ad[]
include::manage/push-threads.ad[]
ifdef::pro[]
include::manage/jvm-properties.ad[]
include::manage/jvm-threads.ad[]
endif::pro[]
include::manage/monitors.ad[]
ifdef::pro[]
include::manage/security.ad[]
include::manage/rest-keys.ad[]
endif::pro[]
include::manage/logging.ad[]