/
advanced-configuration.xml
871 lines (850 loc) · 48 KB
/
advanced-configuration.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
<?xml version="1.0" encoding="UTF-8"?>
<chapter version="5.0" xml:id="advanced-configuration" xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns:ns="http://docbook.org/ns/docbook"
xmlns:mml="http://www.w3.org/1998/Math/MathML"
xmlns:html="http://www.w3.org/1999/xhtml">
<title>Advanced Configuration</title>
<section id='controlling-synchronization'>
<title>Controlling Synchronization</title>
<para>
The frequency of data synchronization is controlled by the coordination of a series of asynchronous events.
</para>
<para>
After data is captured, the first event that occurs is the routing of the
captured <xref linkend="data" xrefstyle="select: title page"/>. Data is routed by the <emphasis>Route Job</emphasis>. It is a single background task that inserts into
<xref linkend="data_event" xrefstyle="select: title page"/> and <xref linkend="outgoing_batch" xrefstyle="select: title page"/>. The Route Job determines which nodes
data will be sent to, as well as how much data will be batched together for transport. When the <literal>start.route.job</literal> SymmetricDS property is set to
<literal>true</literal>, the frequency that routing occurs is controlled by the <literal>job.routing.period.time.ms</literal>. Each time data is routed, the
<xref linkend="data_ref" xrefstyle="select: title page"/> table is updated with the id of the last contiguous data row to have been processed. This is done so the query to find
unrouted data is optimal.
</para>
<para>
After data is routed, it awaits transport to the target nodes. Transport can occur when a client node is configured to pull data or when the host node is configured to push data. These
events are controlled by the <emphasis>Push</emphasis> and the <emphasis>Pull Jobs</emphasis>. When the <literal>start.pull.job</literal> SymmetricDS property is set to
<literal>true</literal>, the frequency that data is pulled is controlled by the <literal>job.pull.period.time.ms</literal>. When the <literal>start.push.job</literal> SymmetricDS property is set to
<literal>true</literal>, the frequency that data is pushed is controlled by the <literal>job.push.period.time.ms</literal>. Data is extracted by channel from the source database's
<xref linkend="data" xrefstyle="select: title page"/> table at an interval controlled by the <literal>extract_period_millis</literal> column on the
<xref linkend="channel" xrefstyle="select: title page"/> table. The <literal>last_extract_time</literal> is
always recorded, by channel, on the <xref linkend="node_channel_control" xrefstyle="select: title page"/> table for the host node's id. When the Pull and Push Job run, if the extract period
has not passed according to the last extract time, then the channel will be skipped for this run. If the <literal>extract_period_millis</literal> is set to zero, data extraction will happen every time the jobs run.
</para>
<para>
SymmetricDS also provides the ability to configure windows of time when synchronization is allowed. This is done using the
<xref linkend="node_group_channel_window" xrefstyle="select: title page"/>
table. A list of allowed time windows can be specified for a node group and a channel. If one or more windows exist, then data will only be extracted and transported if the time
of day falls within the window of time specified. The configured times are always for the target node's local time. If the <literal>start_time</literal> is greater than the <literal>end_time</literal>, then the window crosses
over to the next day.
</para>
<para>
All data loading may be disabled by setting the <literal>dataloader.enable</literal> property to false. This has the effect of not allowing incoming synchronizations, while allowing outgoing
synchronizations. All data extractions may be disabled by setting the <literal>dataextractor.enable</literal> property to false. These properties can be controlled by inserting into the
root server's <xref linkend="parameter" xrefstyle="select: title page"/> table. These properties affect every channel with the exception of the 'config' channel.
</para>
</section>
<section id="multi-tier">
<title>Multi-Tiered Synchronization</title>
<para>
As shown <link linkend="terminology-node-organization">previously</link>, at times there may be
scenarios where data needs to flow through multiple tiers of nodes that
are organized in a tree-like network with each tier requiring a different subset of data. For example,
you may have a system where the lowest tier may by a computer or device located in a store. Those devices
may connect to a server located physically at that store. Then the store server may communicate with
a corporate server for example. In this case, the three tiers would be device, store, and corporate.
Each tier is typically represented by a node group. Each node in
the tier would belong to the node group representing that tier.
</para>
<para>
A node will always push and pull data to other node groups according to the node group link configuration.
A node can only pull and push data to other nodes that are represented <literal>node</literal> table in its database and
having <literal>sync_enabled = 1</literal>.
Because of this, a tree-like
hierarchy of nodes can be created by having only a subset of nodes belonging to the same node group represented at the different branches of the tree.
</para>
<para>
If auto registration is turned <emphasis>off</emphasis>, then this setup must occur manually by opening registration
for the desired nodes at the desired parent node and by configuring each node's <literal>registration.url</literal>
to be the parent node's URL.
The parent node is always tracked by the setting of the parent's <literal>node_id</literal> in the <literal>created_at_node_id</literal> column of the new node.
When a node registers and downloads its configuration it is always provided the configuration for nodes
that might register with the node itself based on the Node Group Links defined in the parent node.
</para>
</section>
<section id='registration-redirect'>
<title>Registration Redirect</title>
<para>
When deploying a multi-tiered system it may be advantageous to have only one registration server, even though the parent node of a registering node
could be any of a number of nodes in the system. In SymmetricDS the parent node is always the node that a child registers with. The
<xref linkend="registration_redirect" xrefstyle="select: title page"/> table allows a single node, usually the root server in the network, to
redirect registering nodes to their true parents. It does so based on a mapping found in the table of the external id (<literal>registrant_external_id</literal>) to the parent's node
id (<literal>registration_node_id</literal>).
</para>
<para>
For example, if it is desired to have a series of regional servers that workstations at retail stores get assigned to based on their <literal>external_id</literal>, the store number, then
you might insert into <xref linkend="registration_redirect" xrefstyle="select: title page"/> the store number as the <literal>registrant_external_id</literal> and the <literal>node_id</literal> of
the assigned region as the <literal>registration_node_id</literal>. When a workstation at the store registers, the root server send an HTTP redirect to the <literal>sync_url</literal> of the node
that matches the <literal>registration_node_id</literal>.
</para>
</section>
<section id="bi-direction-sync">
<title>Bi-Directional Synchronization</title>
<para>
SymmetricDS allows tables to be synchronized bi-directionally. Note that an outgoing
synchronization does not process changes during an incoming synchronization on the same node unless the trigger
was created with the <literal>sync_on_incoming_batch</literal> flag set. If the <literal>sync_on_incoming_batch</literal> flag
is set, then update loops are prevented by a feature that is available in most database dialects.
More specifically, during an incoming synchronization the source <literal>node_id</literal> is put into a database session variable that is
available to the database trigger. Data events are not generated if the target <literal>node_id</literal>
on an outgoing synchronization is equal to the source <literal>node_id</literal>.
</para>
<para>
By default, only the columns that changed will be updated in the target system.
</para>
<para>
More complex conflict resolution strategies can be accomplished by using the
<literal>IDataLoaderFilter</literal> extension point which has access to both
old and new data.
</para>
</section>
<section id="dead-triggers">
<title>Dead Triggers</title>
<para>
Normally a Trigger is specified to capture data changes to a table and send them to
a target Node Group. A "dead" Trigger is one that does not capture data changes.
In other words, the <literal>sync_on_insert</literal>, <literal>sync_on_update</literal>, and <literal>sync_on_delete</literal> properties
for the Trigger are all set to false. However, because the Trigger is specified,
it <emphasis>will</emphasis>
be included in the initial load of data for target Nodes.
</para>
<para>
A dead Trigger might be used to load a read-only lookup table, for example. It could also
be used
to load a table that needs populated with example or default data.
Another use is a recovery load of data for tables that have a single direction
of synchronization. For example, a retail store records sales transaction that
synchronize in one direction by trickling back to the central office.
If the retail store needs to recover all the sales transactions from the central office,
they can be sent
are part of an initial load from the central office by setting up dead Triggers
that "sync" in that direction.
</para>
<para>
The following SQL statement sets up a non-syncing dead Trigger that sends
the <literal>sale_transaction</literal> table to the "store" Node Group from the "corp" Node Group during
an initial load.
<programlisting>
<![CDATA[
insert into sym_trigger (TRIGGER_ID,SOURCE_CATALOG_NAME,
SOURCE_SCHEMA_NAME,SOURCE_TABLE_NAME,CHANNEL_ID,
SYNC_ON_UPDATE,SYNC_ON_INSERT,SYNC_ON_DELETE,
SYNC_ON_INCOMING_BATCH,NAME_FOR_UPDATE_TRIGGER,
NAME_FOR_INSERT_TRIGGER,NAME_FOR_DELETE_TRIGGER,
SYNC_ON_UPDATE_CONDITION,SYNC_ON_INSERT_CONDITION,
SYNC_ON_DELETE_CONDITION,EXTERNAL_SELECT,
TX_ID_EXPRESSION,EXCLUDED_COLUMN_NAMES,
CREATE_TIME,LAST_UPDATE_BY,LAST_UPDATE_TIME)
values ('SALE_TRANSACTION_DEAD',null,null,
'SALE_TRANSACTION','transaction',
0,0,0,0,null,null,null,null,null,null,null,null,null,
current_timestamp,'demo',current_timestamp);
insert into sym_router (ROUTER_ID,TARGET_CATALOG_NAME,TARGET_SCHEMA_NAME,
TARGET_TABLE_NAME,SOURCE_NODE_GROUP_ID,TARGET_NODE_GROUP_ID,ROUTER_TYPE,
ROUTER_EXPRESSION,SYNC_ON_UPDATE,SYNC_ON_INSERT,SYNC_ON_DELETE,
CREATE_TIME,LAST_UPDATE_BY,LAST_UPDATE_TIME)
values ('CORP_2_STORE',null,null,null,
'corp','store',null,null,1,1,1,
current_timestamp,'demo',current_timestamp);
insert into sym_trigger_router (TRIGGER_ID,ROUTER_ID,INITIAL_LOAD_ORDER,
INITIAL_LOAD_SELECT,CREATE_TIME,LAST_UPDATE_BY,LAST_UPDATE_TIME)
values ('SALE_TRANSACTION_DEAD','CORP_2_REGION',100,null,
current_timestamp,'demo',current_timestamp);
]]></programlisting>
</para>
</section>
<section id='jms-publishing'>
<title>JMS Publishing</title>
<para>
With the proper configuration SymmetricDS can publish XML messages of captured data changes to
JMS during routing or transactionally while data loading synchronized data into a target database.
The following explains how to publish to JMS during synchronization to the target database.
</para>
<para>
The XmlPublisherDataLoaderFilter is a
<link linkend="extensions-data-loader-filter">IDataLoaderFilter</link> that may be configured to
publish specific tables as an XML message to a JMS provider.
See <link linkend="extensions">extension points</link> for information on how
to configure an extension point. If the publish to JMS fails, the batch will be marked in error,
the loaded data for the batch will be rolled back
and the batch will be retried during the next synchronization run.
</para>
<para>
The following is an example extension point configuration that will publish four tables in XML with a root
tag of <emphasis>'sale'</emphasis>. Each XML message will be grouped by the batch and the column names identified by
the groupByColumnNames property which have the same values.
<programlisting><![CDATA[<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:context="http://www.springframework.org/schema/context"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context-3.0.xsd">
<bean id="publishingFilter"
class="org.jumpmind.symmetric.integrate.XmlPublisherDataLoaderFilter">
<property name="xmlTagNameToUseForGroup" value="sale"/>
<property name="tableNamesToPublishAsGroup">
<list>
<value>SALE_TX</value>
<value>SALE_LINE_ITEM</value>
<value>SALE_TAX</value>
<value>SALE_TOTAL</value>
</list>
</property>
<property name="groupByColumnNames">
<list>
<value>STORE_ID</value>
<value>BUSINESS_DAY</value>
<value>WORKSTATION_ID</value>
<value>TRANSACTION_ID</value>
</list>
</property>
<property name="publisher">
<bean class="org.jumpmind.symmetric.integrate.SimpleJmsPublisher">
<property name="jmsTemplate" ref="definedSpringJmsTemplate"/>
</bean>
</property>
</bean>
</beans>]]></programlisting>
</para>
<para>
The publisher property on the XmlPublisherDataLoaderFilter takes an interface of type IPublisher. The implementation
demonstrated here is an implementation that publishes to JMS using Spring's
<ulink url="http://static.springsource.org/spring/docs/3.0.x/spring-framework-reference/html/jms.html#jms-jmstemplate">JMS template</ulink>.
Other implementations of IPublisher could easily publish the XML to other targets like an HTTP server, the file system or secure copy it to another server.
</para>
<para>
The above configuration will publish XML similiar to the following:
<programlisting><![CDATA[<?xml version="1.0" encoding="UTF-8"?>
<sale xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
id="0012010-01-220031234" nodeid="00001" time="1264187704155">
<row entity="SALE_TX" dml="I">
<data key="STORE_ID">001</data>
<data key="BUSINESS_DAY">2010-01-22</data>
<data key="WORKSTATION_ID">003</data>
<data key="TRANSACTION_ID">1234</data>
<data key="CASHIER_ID">010110</data>
</row>
<row entity="SALE_LINE_ITEM" dml="I">
<data key="STORE_ID">001</data>
<data key="BUSINESS_DAY">2010-01-22</data>
<data key="WORKSTATION_ID">003</data>
<data key="TRANSACTION_ID">1234</data>
<data key="SKU">9999999</data>
<data key="PRICE">10.00</data>
<data key="DESC" xsi:nil="true"/>
</row>
<row entity="SALE_LINE_ITEM" dml="I">
<data key="STORE_ID">001</data>
<data key="BUSINESS_DAY">2010-01-22</data>
<data key="WORKSTATION_ID">003</data>
<data key="TRANSACTION_ID">1234</data>
<data key="SKU">9999999</data>
<data key="PRICE">10.00</data>
<data key="DESC" xsi:nil="true"/>
</row>
<row entity="SALE_TAX" dml="I">
<data key="STORE_ID">001</data>
<data key="BUSINESS_DAY">2010-01-22</data>
<data key="WORKSTATION_ID">003</data>
<data key="TRANSACTION_ID">1234</data>
<data key="AMOUNT">1.33</data>
</row>
<row entity="SALE_TOTAL" dml="I">
<data key="STORE_ID">001</data>
<data key="BUSINESS_DAY">2010-01-22</data>
<data key="WORKSTATION_ID">003</data>
<data key="TRANSACTION_ID">1234</data>
<data key="AMOUNT">21.33</data>
</row>
</sale>
]]></programlisting>
To publish JMS messages during routing
the same pattern is valid, with the exception that the extension point would be the XmlPublisherDataRouter and the router
would be configured by setting the <literal>router_type</literal> of a <xref linkend="router" xrefstyle="select: title page"/> to the Spring bean
name of the registered extension point. Of course, the router would need to be linked through <xref linkend="trigger_router" xrefstyle="select: title page"/>s
to each <xref linkend="trigger" xrefstyle="select: title page"/> table that needs published.
</para>
</section>
<section id="purge">
<title>Database Purging</title>
<para>
Purging is the act of cleaning up captured data that is no longer needed in SymmetricDS's runtime tables.
Data is purged through
delete statements by the <emphasis>Purge Job</emphasis>. Only data that has been successfully synchronized will be purged. Purged tables include:
<itemizedlist>
<listitem>
<xref linkend="data" xrefstyle="select: title page"/>
</listitem>
<listitem>
<xref linkend="data_event" xrefstyle="select: title page"/>
</listitem>
<listitem>
<xref linkend="outgoing_batch" xrefstyle="select: title page"/>
</listitem>
<listitem>
<xref linkend="incoming_batch" xrefstyle="select: title page"/>
</listitem>
<listitem>
<xref linkend="statistic" xrefstyle="select: title page"/>
</listitem>
</itemizedlist>
The purge job is enabled by the <literal>start.purge.job</literal> SymmetricDS property. The job runs periodically according to the
<literal>job.purge.period.time.ms</literal> property. The default period is to run every ten minutes.
</para>
<para>
Two retention period properties
indicate how much history SymmetricDS will retain before purging. The <literal>purge.retention.minutes</literal> property indicates the period
of history to keep for synchronization tables. The default value is 5 days.
The <literal>statistic.retention.minutes</literal> property
indicates the period of history to keep for statistics. The default value is also 5 days.
</para>
<para>
The purge properties should be adjusted according to how much data is flowing through the system and the amount of storage space the database has.
For an initial deployment it is recommended that the purge properties be kept at the defaults, since it is often helpful to be able to look at
the captured data in order to triage problems and profile the synchronization patterns. When scaling up to more nodes, it is recomended that the
purge parameters be scaled back to 24 hours or less.
</para>
</section>
<section id="temporary-files">
<title>Temporary Files</title>
<para>
SymmetricDS creates temporary extraction and data load files with the CSV payload of a synchronization when
the value of the <literal>stream.to.file.threshold.bytes</literal> SymmetricDS property has been reached. Before reaching the threshold, files
are streamed to/from memory. The default threshold value is 32,767 bytes. This feature may be turned off by setting the <literal>stream.to.file.enabled</literal>
property to false.
</para>
<para>
SymmetricDS creates these temporary files in the directory specified by the <literal>java.io.tmpdir</literal> Java System property. When
SymmmetricDS starts up, stranded temporary files are aways cleaned up. Files will only be stranded if the SymmetricDS engine is force killed.
</para>
<para>
The location of the temporary directory may be changed by setting the Java System property passed into the Java program at startup. For example,
<programlisting>
-Djava.io.tmpdir=/home/.symmetricds/tmp
</programlisting>
</para>
</section>
<section id="clustering">
<title>Clustering</title>
<para>
A single SymmetricDS node may be clustered across a series of instances, creating a web farm. A node might be clustered to provide load balancing and failover, for example.
</para>
<para>
When clustered, a hardware load balancer is typically used
to round robin client requests to the cluster. The load balancer should be configured for stateless connections.
Also, the <link linkend='basic-properties'>sync.url</link>
SymmetricDS property should be set to the URL of the load balancer.
</para>
<para>
If the cluster will be running any of the SymmetricDS jobs, then the <literal>cluster.lock.enabled</literal> property should be set to <literal>true</literal>.
By setting this property to true, SymmetricDS will use a row in the <link linkend="lock">lock</link> table as a semaphore to make sure that only one instance at a time
runs a job. When a lock is acquired, a row is updated in the lock table with the time of the lock and the server id of the locking job. The lock time is set back to null
when the job is finished running. Another instance of SymmetricDS cannot aquire a lock until the locking instance (according to the server id) releases the lock. If an
instance is terminated while the lock is still held, an instance with the same server id is allowed to reaquire the lock. If the locking instance remains down, the lock can be
broken after a period of time, specified by the <literal>cluster.lock.timeout.ms</literal> property, has expired. Note that if the job is still running and the lock
expires, two jobs could be running at the same time which could cause database deadlocks.
</para>
<para>
By default, the locking server id is the hostname of the server. If two clustered instances are running on the same server, then the <literal>cluster.server.id</literal> property
may be set to indicate the name that the instance should use for its server id.
</para>
<para>
When deploying SymmetricDS to an application server like Tomcat or JBoss, no special session clustering needs to be configured for the application server.
</para>
</section>
<section id='logging'>
<title>Logging Configuration</title>
<para>
The standalone SymmetricDS installation uses <ulink url="http://logging.apache.org/log4j/1.2/index.html">Log4J</ulink> for logging. The configuration file is <literal>conf/log4j.xml</literal>.
The <literal>log4j.xml</literal> file has hints as to what logging can be enabled for useful, finer-grained logging.
</para>
<para>
SymmetricDS proxies all of its logging through <ulink url="http://commons.apache.org/logging/">Commons Logging</ulink>. When deploying to an application server, if Log4J is not
being leveraged, then the general rules for for Commons Logging apply.
</para>
</section>
<section id="extensions">
<title>Extension Points</title>
<para>
SymmetricDS may be extended via a plug-in like architecture where extension point interfaces
may be implemented by a custom class and registered with the synchronization engine. All supported extension
points extend the IExtensionPoint interface. The currently available extension points are documented in the following sections.
</para>
<para>
When the synchronization engine starts up, a Spring
post processor searches the Spring ApplicationContext for any registered classes which implement
IExtensionPoint. An IExtensionPoint designates whether it should be auto registered or not. If the extension point
is to be auto registered then the post processor registers the known interface with the appropriate service.
</para>
<para>
The INodeGroupExtensionPoint interface may be optionally implemented to designate that auto registered
extension points should only be auto registered with specific node groups.
<programlisting><![CDATA[/**
* Only apply this extension point to the 'root' node group.
*/
public String[] getNodeGroupIdsToApplyTo() {
return new String[] { "root" };
}
]]></programlisting>
</para>
<para>
SymmetricDS will look for Spring configured extensions in the application Classpath by importing any Spring
XML configuration files found matching the following pattern: <literal>META-INF/services/symmetric-*-ext.xml</literal>. When packaged in
a jar file the <literal>META-INF</literal> directory should be at the root of the jar file. When packaged in a war file, the <literal>META-INF</literal> directory
should be in the <literal>WEB-INF/classes</literal> directory.
</para>
<section id="parameter-filter">
<title>IParameterFilter</title>
<para>
Parameter values can be specified in code using a parameter filter. Note that there can be only one parameter
filter per engine instance. The IParameterFilter replaces the deprecated IRuntimeConfig from prior releases.
<programlisting><![CDATA[public class MyParameterFilter
implements IParameterFilter, INodeGroupExtensionPoint {
/**
* Only apply this filter to stores
*/
public String[] getNodeGroupIdsToApplyTo() {
return new String[] { "store" };
}
public String filterParameter(String key, String value) {
// look up a store number from an already existing properties file.
if (key.equals(ParameterConstants.EXTERNAL_ID)) {
return StoreProperties.getStoreProperties().
getProperty(StoreProperties.STORE_NUMBER);
}
return value;
}
public boolean isAutoRegister() {
return true;
}
}]]></programlisting>
</para>
</section>
<section id="extensions-data-loader-filter">
<title>IDataLoaderFilter</title>
<para>
Data can be filtered as it is loaded into the target database. It can also be
filtered when
it is extracted from the source database.
As data is loaded into the target database, a filter can change the
data in a column or save it somewhere else. It can also specify by the
return value of the function call that the data loader should continue on
and load the data (by returning true) or ignore it (by returning false). One
possible use of the filter might be to
route credit card data to a secure database and blank it out as it loads
into a less-restricted reporting database.
</para>
<para>
An IDataLoaderContext is passed to each of the callback methods. A new
context is created for each synchronization. The context provides methods to
lookup column indexes by column name, get table meta data, and access to
old data if the <literal>sync_column_level</literal> flag is enabled. The context also provides a means
to share data during a synchronization between different rows of data that are
committed in a database transaction and are in the same channel. It does so by
providing a context cache which can be populated by the extension point.
</para>
<para>
Many times the IDataLoaderFilter will be combined with the IBatchListener. The
XmlPublisherFilter (in the <literal>org.jumpmind.symmetric.ext</literal> package) is a good example
of using the combination of the two extension points in order to create XML messages
to be published to JMS.
</para>
<para>
A class implementing the IDataLoaderFilter interface is injected onto the
DataLoaderService in order to receive callbacks when data is inserted,
updated, or deleted.
<programlisting><![CDATA[public MyFilter implements IDataLoaderFilter {
public boolean isAutoRegister() {
return true;
}
public boolean filterInsert(IDataLoaderContext context,
String[] columnValues) {
return true;
}
public boolean filterUpdate(IDataLoaderContext context,
String[] columnValues, String[] keyValues) {
return true;
}
public void filterDelete(IDataLoaderContext context,
String[] keyValues) {
return true;
}
}]]></programlisting>
</para>
<para>
The filter class is specified as a Spring-managed bean. A custom Spring XML file
is specified as follows in a jar at <literal>META-INF/services/symmetric-myfilter-ext.xml</literal>.
<programlisting><![CDATA[<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:context="http://www.springframework.org/schema/context"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context-3.0.xsd">
<bean id="myFilter" class="com.mydomain.MyFilter"/>
</beans>]]></programlisting>
</para>
</section>
<section id="extensions-table-column-filter">
<title>ITableColumnFilter</title>
<para>
Implement this extension point to filter out specific columns from
use by the dataloader. Only one column filter may be added per target table.
</para>
</section>
<section id="extensions-batch-listener">
<title>IBatchListener</title>
<para>
This extension point is called whenever a batch has completed loading but before
the transaction has committed.
</para>
</section>
<section id="extensions-acknowledge-event-listener">
<title>IAcknowledgeEventListener</title>
<para>
Implement this extension point to receive callback events when a batch is acknowledged.
The callback for this listener happens at the point of extraction.
</para>
</section>
<section id="extensions-reload-listener">
<title>IReloadListener</title>
<para>
Implement this extension point to listen in and take
action before or after a reload is requested for a Node. The callback for this listener
happens at the point of extraction.
</para>
</section>
<section id="extensions-extractor-filter">
<title>IExtractorFilter</title>
<para>
This extension point is called after data has been extracted, but before it has been streamed. It
has the ability to inspect each row of data to take some action and indicate, if necessary, that the
row should not be streamed.
</para>
</section>
<section id="extensions-sync-url-extension">
<title>ISyncUrlExtension</title>
<para>
This extension point is used to select an appropriate URL based on
the URI provided in the <literal>sync_url</literal> column of <literal>sym_node</literal>.
</para>
<para>
To use this extension point configure the sync_url for a node with the
protocol of ext://beanName. The beanName is the name you give the extension
point in the extension xml file.
</para>
</section>
<section id="extensions-node-id-generator">
<title>INodeIdGenerator</title>
<para>
This extension point allows SymmetricDS users to implement their own algorithms for how
node ids and passwords are generated or selected during the registration process. There may be
only one node generator per SymmetricDS instance.
</para>
</section>
<section id="extensions-trigger-creation-listener">
<title>ITriggerCreationListener</title>
<para>
Implement this extension point to get status callbacks during trigger creation.
</para>
</section>
<section id="extensions-batch-algorithm">
<title>IBatchAlgorithm</title>
<para>
Implement this extension point and set the name of the Spring bean on the batch_algorithm column of the Channel table to use.
This extension point gives fine grained control over how a channel is batched.
</para>
</section>
<section id="extensions-data-router">
<title>IDataRouter</title>
<para>
Implement this extension point and set the name of the Spring bean on the router_type column
of the Router table to use. This extension point gives the ability to programatically decide
which nodes data should be routed to.
</para>
</section>
<section id="extensions-heartbeat-listener">
<title>IHeartbeatListener</title>
<para>
Implement this extension point to get callbacks during the heartbeat job.
</para>
</section>
<section id="extensions-offline-client-listener">
<title>IOfflineClientListener</title>
<para>
Implement this extension point to get callbacks for offline events on client nodes.
</para>
</section>
<section id="extensions-offline-server-listener">
<title>IOfflineServerListener</title>
<para>
Implement this extension point to get callbacks for offline events detected on a server node during monitoring of client nodes.
</para>
</section>
<section id="extensions-node-password">
<title>INodePasswordFilter</title>
<para>
Implement this extension point to intercept the saving and rendering of the node password.
</para>
</section>
<section id="extensions-servlets">
<title>IServletExtension</title>
<para>
Implement this extension point to allow additional Servlets to be registered with SymmetricDS. This is probably only
useful if SymmetricDS is running in standalone or embedded mode.
</para>
</section>
</section>
<section id="encrypted-passwords">
<title>Encrypted Passwords</title>
<para>
The <literal>db.user</literal> and <literal>db.password properties</literal> will accept encrypted text, which protects
against casual observation. The text is prefixed with <literal>enc:</literal> to indicate
that it is encrypted. To encrypt text, use the following command:
</para>
<para>
<command>sym -e secret</command>
</para>
<para>
The text is encrypted by the cipher defined as alias "sym.secret" in the
Java keystore. The keystore is specified by the "sym.keystore.file" system property, which
defaults to <filename class="filename">security/keystore</filename>.
If a cipher is not found, a default cipher using Triple DES with a random password
is generated.
</para>
</section>
<section id="secure-transport">
<title>Secure Transport</title>
<para>
By specifying the "https" protocol for a URL, SymmetricDS will communicate over
Secure Sockets Layer (SSL) for an encrypted transport. The following properties
need to be set with "https" in the URL:
<variablelist>
<varlistentry>
<term>
<command>sync.url</command>
</term>
<listitem>
<para>
This is the URL of the current node, so if you want to force other
nodes to communicate over SSL with this node, you specify "https" in the URL.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>
<command>registration.url</command>
</term>
<listitem>
<para>
This is the URL where the node will connect for registration when it
first starts up. To protect the registration with SSL, you specify
"https" in the URL.
</para>
</listitem>
</varlistentry>
</variablelist>
For incoming HTTPS connections, SymmetricDS depends on the webserver where
it is deployed, so the webserver must be configured for HTTPS.
As a standalone deployment, the "sym" launcher command provides options for
enabling HTTPS support.
</para>
<section id="secure-transport-sym">
<title>Sym Launcher</title>
<para>
The "sym" launch command uses Jetty as an embedded web server.
Using command line options, the web server can be told to listen for
HTTP, HTTPS, or both.
</para>
<para>
<command>sym --port 8080 --server</command>
</para>
<para>
<command>sym --secure-port 8443 --secure-server</command>
</para>
<para>
<command>sym --port 8080 --secure-port 8443 --mixed-server</command>
</para>
</section>
<section id="secure-transport-tomcat">
<title>Tomcat</title>
<para>
If you deploy SymmetricDS to Apache Tomcat, it can be secured by editing the
<filename class="filename">TOMCAT_HOME/conf/server.xml</filename>
configuration file. There is already a line that can be uncommented
and changed to the following:
<programlisting>
<![CDATA[<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
maxThreads="150" scheme="https" secure="true"
clientAuth="false" sslProtocol="TLS"
keystoreFile="/symmetric-ds-1.x.x/security/keystore" />]]></programlisting>
</para>
</section>
<section id="secure-transport-keystore">
<title>Keystores</title>
<para>
When SymmetricDS connects to a URL with HTTPS, Java checks the validity of the
certificate using the built-in trusted keystore located at
<filename class="filename">JRE_HOME/lib/security/cacerts</filename>.
The "sym" launcher command overrides the trusted keystore to use its own
trusted keystore instead, which is located at
<filename class="filename">security/cacerts</filename>.
This keystore contains the certificate aliased as "sym" for use in testing
and easing deployments.
The trusted keystore can be overridden
by specifying the <literal>javax.net.ssl.trustStore</literal> system property.
</para>
<para>
When SymmetricDS is run as a secure server with the "sym" launcher,
it accepts incoming requests using the key installed in the keystore
located at
<filename class="filename">security/keystore</filename>.
The default key is provided for convenience of testing, but should be
re-generated for security.
</para>
</section>
<section id="secure-transport-keys">
<title>Generating Keys</title>
<para>
To generate new keys and install a server certificate, use the
following steps:
</para>
<procedure>
<step>
<para>
Open a command prompt and navigate to the
<filename class="directory">security</filename>
subdirectory of your SymmetricDS installation.
</para>
</step>
<step>
<para>Delete the old key pair and certificate.</para>
<para>
<command>keytool -keystore keystore -delete -alias sym</command>
</para>
<para>
<command>keytool -keystore cacerts -delete -alias sym</command>
</para>
<programlisting><![CDATA[Enter keystore password: changeit]]></programlisting>
</step>
<step>
<para>Generate a new key pair.</para>
<para>
<command>keytool -keystore keystore -alias sym -genkey -keyalg RSA -validity 10950</command>
</para>
<programlisting>
<![CDATA[Enter keystore password: changeit
What is your first and last name?
[Unknown]: localhost
What is the name of your organizational unit?
[Unknown]: SymmetricDS
What is the name of your organization?
[Unknown]: JumpMind
What is the name of your City or Locality?
[Unknown]:
What is the name of your State or Province?
[Unknown]:
What is the two-letter country code for this unit?
[Unknown]:
Is CN=localhost, OU=SymmetricDS, O=JumpMind, L=Unknown, ST=Unknown, C=Unknown
correct?
[no]: yes
Enter key password for <sym>
(RETURN if same as keystore password):]]></programlisting>
</step>
<step>
<para>Export the certificate from the private keystore.</para>
<para>
<command>keytool -keystore keystore -export -alias sym -rfc -file sym.cer</command>
</para>
</step>
<step>
<para>Install the certificate in the trusted keystore.</para>
<para>
<command>keytool -keystore cacerts -import -alias sym -file sym.cer</command>
</para>
</step>
</procedure>
</section>
</section>
<section id="basic-auth">
<title>Basic Authentication</title>
<para>
SymmetricDS supports basic authentication for client and server nodes.
To configure a client node to use basic authentication when communicating with a server node,
specify the following startup parameters:
</para>
<variablelist>
<varlistentry>
<term>
<command>http.basic.auth.username</command>
</term>
<listitem>
<para>
username for client node basic authentication.
[ Default: ]
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>
<command>http.basic.auth.password</command>
</term>
<listitem>
<para>
password for client node basic authentication.
[ Default: ]
</para>
</listitem>
</varlistentry>
</variablelist>
<para>
The SymmetricDS Standalone and Embedded Server also support basic authentication.
This feature is enabled by specifying the basic authentication username and
password using the following startup parameters:
</para>
<variablelist>
<varlistentry>
<term>
<command>embedded.webserver.basic.auth.username</command>
</term>
<listitem>
<para>
username for basic authentication for an embedded server
or standalone server node.
[ Default: ]
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>
<command>embedded.webserver.basic.auth.password</command>
</term>
<listitem>
<para>
password for basic authentication for an embedded server
or standalone server node.
[ Default: ]
</para>
</listitem>
</varlistentry>
</variablelist>
<para>
If the server node is deployed to Tomcat or another application server as a WAR or EAR file, then
basic authentication is setup with the standard configuration in the WEB.xml file.
</para>
</section>
</chapter>