user-guide.xml

<?xml version="1.0"?>
<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
"http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd">
<book>
    <bookinfo>
        <title>SymmetricDS User Guide</title>
        <authorgroup>
            <author>
                <firstname>Eric</firstname>
                <surname>Long</surname>
            </author>
            <author>
                <firstname>Chris</firstname>
                <surname>Henson</surname>
            </author>
        </authorgroup>
        <edition>Version 1.0 for SymmetricDS v1.0.0</edition>
        <copyright>
            <year>2007</year>
            <holder>Eric Long</holder>
            <holder>Chris Henson</holder>
        </copyright>
        <legalnotice>
            <para>
                Permission to use, copy, modify, and distribute the SymmetricDS User Guide Version
                1.0 for any purpose and without fee is hereby granted in perpetuity, provided that
                the above copyright notice and this paragraph appear in all copies.
            </para>
        </legalnotice>
    </bookinfo>
    <preface>
        <title>Preface</title>
        <para>
            SymmetricDS is web-enabled, database independent, data synchronization software. It uses
            web and database technologies to replicate tables between relational databases in near
            real time. The software was designed to scale for a large number of databases, work
            across low-bandwidth connections, and withstand periods of network outage.
        </para>
        <para>
            This User Guide describes the SymmetricDS library for data synchronization. It is
            intended for users who want to be quickly familiarized with the software, configure it,
            and use its features.
        </para>
    </preface>
    <chapter>
        <title>Introduction</title>
        <section>
            <title>The Problem of Synchronization</title>
        </section>
        <section>
            <title>SymmetricDS Feature Overview</title>
            <section>
                <title>Notification Schemes</title>
                <para>Push (trickle-back data) or Pull (trickle-poll data) changes</para>
            </section>
            <section>
                <title>Two-Way Table Synchronization</title>
                <para>
                    The same table can be synchronized both to and from the host system while
                    avoiding update loops
                </para>
            </section>
            <section>
                <title>Data Channels</title>
                <para>Table synchronizations are grouped into independent channels</para>
            </section>
            <section>
                <title>Transaction Awareness</title>
                <para>Data updates are recorded and replayed with the same atomicity</para>
            </section>
            <section>
                <title>Data Filtering and Rerouting</title>
                <para>Allows for localized passwords and sensitive data filtering/routing</para>
            </section>
            <section>
                <title>HTTP Transport</title>
                <para>
                    Pluggable transport defaults to Representation State Transfer (REST-style) HTTP
                    services
                </para>
            </section>
            <section>
                <title>Remote Management</title>
                <para>Administration through a Java Management Extensions (JMX) console</para>
            </section>
        </section>
        <section>
            <title>Requirements</title>
            <para>
                SymmetricDS is written in Java 5 and requires a Java SE Runtime Environment (JRE) or
                Java SE Development Kit (JDK) version 5.0 or above.
            </para>
            <para>
                Any database with trigger technology and a JDBC driver has the potential to run
                SymmetricDS. The database is abstracted through a Database Dialect in order to
                support specific features of each database. The following Database Dialects have
                been included with release 1.0:
            </para>
            <itemizedlist spacing="compact">
                <listitem>
                    <para>MySQL version 5.0.2 and above</para>
                </listitem>
                <listitem>
                    <para>Oracle version 8.1.7 and above</para>
                </listitem>
            </itemizedlist>
        </section>
        <section>
            <title>Background</title>
            <para>
            	While implementing a commercial Point of Sale (POS) system for a large retailer,
            	the development team concluded that the software provided for trickling back transactions
            	to the general office would not meet the project needs.
            	The list of problems in the requirements made finding the ideal solution difficult:
			</para>
            <itemizedlist spacing="compact">
                <listitem>
                    <para>Sending and receiving data with 2000 stores during peak holiday loads.</para>
                </listitem>
                <listitem>
                    <para>Supporting one database platform at the store and another at general office.</para>
                </listitem>
                <listitem>
                    <para>Synchronizing some data in one direction, and other data in both directions.</para>
                </listitem>
                <listitem>
                    <para>Filtering out sensitive data and re-routing it to a protected database.</para>
                </listitem>
                <listitem>
                    <para>Preparing the store database with an initial load of data from general office.</para>
                </listitem>
            </itemizedlist>
			<para>
			    The team created a custom solution that met the requirements and made the project
			    successful.  From this initial challenge came the knowledge and experience that
			    SymmetricDS benefits from today.
            </para>
        </section>
        <section>
            <title>Version Numbering</title>
            <para>
                The software is released with a version number based on the
                <ulink url="http://apr.apache.org/versioning.html">
                    Apache Portable Runtime Project
                </ulink>
                version guidelines. In summary, the version is denoted as three integers in the
                format of MAJOR.MINOR.PATCH. Major versions are incompatible at the API level, and
                they can include any kind of change. Minor versions are compatible with older
                versions at the API and binary level, and they can intoduce new functions or remove
                old ones. Patch versions are perfectly compatible, and they are released to fix
                defects.
            </para>
        </section>
    </chapter>
    <chapter>
        <title>Getting Started</title>
        <para>
            This chapter is a hands-on tutorial that shows how to synchronize the sample database
            between two running nodes of SymmetricDS.
        </para>
        <para>
            To get started, you create a root (central source) database and a client (remote target)
            database that each run the SymmetricDS software to synchronize changes. You open
            registration for the client, which registers with the root to receive the
            synchronization configuration. You ask the root to reload the client in order to perform
            an initial synchronization. Then you make changes to each database and observe the
            changes propagating.
        </para>
        <section>
            <title>Installing SymmetricDS</title>
            <para>
                Install the SymmetricDS software and configure it with your database connection
                information.
            </para>
            <procedure>
                <step>
                    <para>
                        Download the
                        <ulink url="http://sourceforge.net/project/showfiles.php?group_id=206470">
                            symmetric-ds.zip
                        </ulink>
                        file from
                        <ulink url="http://www.symmetricds.org/">http://www.symmetricds.org/</ulink>
                    </para>
                </step>
                <step>
                    <para>
                        Unzip the file, which creates a
                        <filename class="directory">symmetric-ds-1.0.0</filename>
                        subdirectory.
                    </para>
                </step>
                <step>
                    <para>Edit the database properties for the root and client nodes.</para>
                    <substeps>
                        <step>
                            <para>
                                Edit the
                                <filename>samples/root.properties</filename>
                                for the root node.
                            </para>
                        </step>
                        <step>
                            <para>
                                Edit the
                                <filename>samples/client.properties</filename>
                                for the client node.
                            </para>
                        </step>
                    </substeps>
                </step>
                <step>
                    <para>
                        Set the following properties in both files for the corresponding databases:
                    </para>
                    <itemizedlist spacing="compact">
                        <listitem>
                            <para>
                                <literal>db.driver</literal>
                                - The class name for the JDBC Driver
                            </para>
                        </listitem>
                        <listitem>
                            <para>
                                <literal>db.url</literal>
                                - The JDBC URL used to connect to the database
                            </para>
                        </listitem>
                        <listitem>
                            <para>
                                <literal>db.user</literal>
                                - The user to login as who can create and update tables
                            </para>
                        </listitem>
                        <listitem>
                            <para>
                                <literal>db.password</literal>
                                - The password for the user to login as
                            </para>
                        </listitem>
                    </itemizedlist>
                </step>
                <step>
                    <para>
                        Set the following properties in the
                        <filename>client.properties</filename>
                        file:
                    </para>
                    <itemizedlist spacing="compact">
                        <listitem>
                            <para>
                                <literal>symmetric.runtime.registration.url</literal>
                                - The HTTP URL of the root node to contact for registration
                                (http://hostname:8080/sync)
                            </para>
                        </listitem>
                    </itemizedlist>
                </step>
            </procedure>
        </section>
        <section>
            <title>Creating and Populating Your Databases</title>
            <important>
                <para>
                    You must first create the databases for your root and client nodes using the
                    administration tools provided by your database vendor. Make sure the name of the
                    databases you create match the settings in the properties files.
                </para>
            </important>
            <para>
                Create and load the root node tables with sample data and the synchronization
                configuration. Create the tables in the client node database to prepare it.
            </para>
            <procedure>
                <step>
                    <para>
                        Open a command prompt and navigate to the
                        <filename class="directory">samples</filename>
                        subdirectory of your SymmetricDS installation.
                    </para>
                </step>
                <step>
                    <para>Create the sample tables in both the root and client databases.</para>
                    <para>
                        <command>symmetric -p root.properties --run-ddl create_sample.xml</command>
                    </para>
                    <para>
                        <command>
                            symmetric -p client.properties --run-ddl create_sample.xml
                        </command>
                    </para>
                    <para>
                        Note that the command may output warnings about altering foreign keys on the
                        tables, which are safe to ignore.
                    </para>
                </step>
                <step>
                    <para>
                        Create the symmetric tables in the root node database. These tables will
                        contain the configuration for synchronization. The following command uses
                        the auto-creation feature to create all the necessary symmetric system
                        tables.
                    </para>
                    <para>
                        <command>symmetric -p root.properties --auto-create</command>
                    </para>
                </step>
                <step>
                    <para>Load sample data and configuration into the root node database.</para>
                    <para>
                        <command>symmetric -p root.properties --run-sql insert_sample.sql</command>
                    </para>
                </step>
                <step>
                    <para>Verify the databases by logging in and listing the tables.</para>
                </step>
            </procedure>
        </section>
        <section>
            <title>Starting SymmetricDS</title>
            <para>Start the SymmetricDS nodes and observe the logging output.</para>
            <procedure>
                <step>
                    <para>
                        Open a command prompt and navigate to the
                        <filename class="directory">samples</filename>
                        subdirectory of your SymmetricDS installation.
                    </para>
                </step>
                <step>
                    <para>Start the root node server.</para>
                    <para>
                        <command>symmetric -p root.properties --port 8080 --server</command>
                    </para>
                    <para>
                        The root node server starts up and creates all the triggers that were
                        configured by the sample configuration. It listens on port 8080 for
                        synchronization and registration requests.
                    </para>
                </step>
                <step>
                    <para>Start the client node server.</para>
                    <para>
                        <command>symmetric -p client.properties --port 9090 --server</command>
                    </para>
                    <para>
                        The client node server starts up and uses the auto-creation feature to
                        create the symmetric system tables. It begins polling the root node in order
                        to register. Since registration is not yet open, the client node receives an
                        authorization failure (HTTP response of 403).
                    </para>
                </step>
            </procedure>
        </section>
        <section>
            <title>Registering a Node</title>
            <para>
                Open registration for the client node using the root node administration feature.
            </para>
            <procedure>
                <step>
                    <para>
                        Open a command prompt and navigate to the
                        <filename class="directory">samples</filename>
                        subdirectory of your SymmetricDS installation.
                    </para>
                </step>
                <step>
                    <para>Open registration for the client node server.</para>
                    <para>
                        <command>
                            symmetric -p root.properties --open-registration "store,1"
                        </command>
                    </para>
                    <para>
                        The registration is opened for a node group called "store" with an external
                        identifier of "1".
                    </para>
                </step>
                <step>
                    <para>
                        Watch the logging output of the client node to see it successfully register
                        with the root node. The client is configured to attempt registration each
                        minute. Once registered, the root and client are enabled for
                        synchronization.
                    </para>
                </step>
            </procedure>
        </section>
        <section>
            <title>Sending Initial Load</title>
            <para>
                Send an initial load of data to the client node using the root node administration
                feature.
            </para>
            <procedure>
                <step>
                    <para>
                        Open a command prompt and navigate to the
                        <filename class="directory">samples</filename>
                        subdirectory of your SymmetricDS installation.
                    </para>
                </step>
                <step>
                    <para>Send an initial load of data to the client node server.</para>
                    <para>
                        <command>symmetric -p root.properties --reload-node 1</command>
                    </para>
                    <para>
                        With this command, the root node queues up an initial load for the client
                        node that will sent the next time the client performs its pull. The initial
                        load includes data for each table that is configured for synchronization.
                    </para>
                </step>
                <step>
                    <para>
                        Watch the logging output of both nodes to see the data transfer. The client
                        is configured to pull data from the root each minute.
                    </para>
                </step>
            </procedure>
        </section>
        <section>
            <title>Pulling Data</title>
            <para>Change data on the root database, which is pulled by the client database.</para>
            <procedure>
                <step>
                    <para>Open an interactive SQL session with the root database.</para>
                </step>
                <step>
                    <para>Add a new item for sale on the root database</para>
                    <para>
                        <command>
                            insert into item_selling_price (price_id, price) values (55, 0.65);
                        </command>
                    </para>
                    <para>
                        <command>
                            insert into item (item_id, price_id, name) values (110000055, 55, 'Soft
                            Drink');
                        </command>
                    </para>
                    <para>
                        Once the statements are committed, the data change is captured and queued
                        for the client node to pull.
                    </para>
                </step>
                <step>
                    <para>
                        Watch the logging output of both nodes to see the data transfer. The client
                        is configured to pull data from the root each minute.
                    </para>
                </step>
                <step>
                    <para>
                        Verify that the new data arrives in the client database using another
                        interactive SQL session.
                    </para>
                </step>
            </procedure>
        </section>
        <section>
            <title>Pushing Data</title>
            <para>Change data on the client database, which is pushed to the root database.</para>
            <procedure>
                <step>
                    <para>Open an interactive SQL session with the client database.</para>
                </step>
                <step>
                    <para>Add a new sale on the client database</para>
                    <para>
                        <command>
                            insert into transaction (tran_id, store, workstation, day, seq) values
                            (1000, '1', '3', '2007-11-01', 100);
                        </command>
                    </para>
                    <para>
                        <command>
                            insert into sale_return_line_item (tran_id, item_id, price, quantity)
                            values (1000, 110000055, 0.65, 1);
                        </command>
                    </para>
                    <para>
                        Once the statements are committed, the data change is captured and queued
                        for the client node to push.
                    </para>
                </step>
                <step>
                    <para>
                        Watch the logging output of both nodes to see the data transfer. The client
                        is configured to push data to the root each minute.
                    </para>
                </step>
                <step>
                    <para>
                        Verify that the new data arrives in the root database using another
                        interactive SQL session.
                    </para>
                </step>
            </procedure>
        </section>
    </chapter>
    <chapter>
        <title>Concepts</title>
        <section>
            <title>Data Model</title>
        </section>
        <section>
            <title>Node</title>
            <para>
                An instance of SymmetricDS that synchronizes data with one or more nodes. Each node
                has a unique identifier (nodeID) that is used when communicating, as well as a
                domain-specific identifier (externalId) that provides context within the local
                system.
            </para>
        </section>
        <section>
            <title>Node Security</title>
            <para>
                Security features like node passwords and open registration flag are stored in the
                NodeSecurity table.
            </para>
        </section>
        <section>
            <title>Node Group</title>
            <para>
                A category of Nodes that synchronizes data with one or more NodeGroups. A common use
                of NodeGroup is to describe a level in a hierarchy of data synchronization.
            </para>
        </section>
        <section>
            <title>Node Group Link</title>
            <para>
                A source NodeGroup sends its data updates to a target NodeGroup using a pull, push,
                or custom technique.
            </para>
        </section>
        <section>
            <title>Channel</title>
            <para>
                A category of data that can be synchronized independently of other Channels.
                Channels allow control over the type of data flowing and prevents one type of
                synchronization from contending with another.
            </para>
        </section>
        <section>
            <title>Trigger</title>
            <para>
                The database triggers that capture changes in the database are automatically
                generated by SymmetricDS. Configuration of which triggers are generated and how they
                will behave is stored in the Trigger table.
            </para>
        </section>
    </chapter>
    <chapter>
        <title>Architecture</title>
        <section>
            <title>Software Components</title>
        </section>
        <section>
            <title>Deployment Options</title>
        </section>
    </chapter>
    <chapter>
        <title>Basic Configuration</title>
        <section>
            <title>Nodes</title>
        </section>
    </chapter>
    <chapter>
        <title>Advanced Configuration</title>
        <section>
            <title>Dead Triggers</title>
        </section>
        <section>
            <title>Auto Creation</title>
        </section>
        <section>
            <title>Secure Transport</title>
        </section>
    </chapter>
    <chapter>
        <title>Administration</title>
        <section>
            <title>Rebuilding Triggers</title>
        </section>
        <section>
            <title>Opening Registration</title>
        </section>
        <section>
            <title>Enabling and Disabling Synchronization</title>
        </section>
        <section>
            <title>Viewing Batches</title>
        </section>
    </chapter>
    <appendix>
        <title>SymmetricDS Data Format</title>
        <para>
            The SymmetricDS Data Format is used to stream data from node to another. The data format
            reader and writer are pluggable with an initial implementation using a format based on
            Comma Separated Values (CSV). Each line in the stream is a record with fields separated
            by commas. String fields are surrounded with double quotes. Double quotes and
            backslashes used in a string field are escaped with a backslash. Binary values are
            represented as a string with hex values in "\0xab" format. The absence of any value in
            the field indicates a null value. Extra spacing is ignored and lines starting with a
            hash are ignored.
        </para>
        <para>
            The first field of each line gives the directive for the line. The following directives
            are used:

            <itemizedlist spacing="compact">
                <listitem>
                    <para>
                        <command>version {major},{minor},{patch}</command>
                    </para>
                    Indicates the version of the file format
                </listitem>
                <listitem>
                    <para>
                        <command>table {table name}</command>
                    </para>
                    Sets the context of which table to operate on
                </listitem>
                <listitem>
                    <para>
                        <command>keys {column name...}</command>
                    </para>
                    Lists the column names that are used as the primary key for the table. Only
                    needs to occur after the first occurrence of the table.
                </listitem>
                <listitem>
                    <para>
                        <command>columns {column name...}</command>
                    </para>
                    Lists all the column names of the table. Only needs to occur after the first
                    occurrence of the table.
                </listitem>
                <listitem>
                    <para>
                        <command>insert {column value...}</command>
                    </para>
                    Insert into the table with the values that correspond with the columns
                </listitem>
                <listitem>
                    <para>
                        <command>update {old key value...},{new column value...}</command>
                    </para>
                    Update the table using the old key values to set the new column values
                </listitem>
                <listitem>
                    <para>
                        <command>delete {old key value...}</command>
                    </para>
                    Delete from the table using the old key values
                </listitem>
            </itemizedlist>
        </para>
        <example>
            <title>Data Format Stream</title>
            <programlisting>
                <![CDATA[
version, 1,0,0
table, item_selling_price
keys, price_id
columns, price, cost
insert, 55, 0.65, 0.55
table, item
keys, item_id 
columns, item_id, price_id, name
insert, 110000055, 55, "Soft Drink"
delete, 110000001
table, item_selling_price
update, 55, 55, 0.75, 0.65
                ]]>
            </programlisting>
        </example>
    </appendix>
</book>