0004943: Fixed some warnings that appeared when running generateDocs

symmetric-assemble/src/asciidoc/appendix.ad

@@ -63,6 +63,9 @@ include::generated/server-parameters.ad[]
 
 [appendix]
 == Database Notes
+
+This section describes specific settings and notes for using each supported database platform.
+
 include::appendix/databases.ad[]
 
 

symmetric-assemble/src/asciidoc/appendix/databases.ad

@@ -1,6 +1,3 @@
-
-This section describes specific settings and notes for using each supported database platform.
-
 [id="databases-compatibility",reftext="Database Compatibility"]
 === Compatibility
 

symmetric-assemble/src/asciidoc/configuration/export.ad

@@ -1,5 +1,5 @@
 
-==== Export
+=== Export
 
 Export your configuration at any time through the web console.   Use the drop down to select the format you would like to use for the export.
 

symmetric-assemble/src/asciidoc/configuration/import.ad

@@ -1,5 +1,5 @@
 
-==== Import
+=== Import
 
 There are two ways a SymmetricDS configuration can be imported.  Both ways support importing an export that is in CSV format.  
 In order to import in SQL format you would run the SQL script via database specific tools or import through the web console in Configure -> Overview.

symmetric-assemble/src/asciidoc/configuration/table-triggers/external-select.ad

@@ -13,23 +13,31 @@ IMPORTANT: The external select SQL must return a single row, single column
 
 ifdef::pro[]
 .Sample External Select SQL that returns STORE_ID based on the ORDER_ID captured in the trigger.
-[source, SQL]
 ====
+
+[source, SQL]
+----
 select STORE_ID 
 from order 
 where order_id=$(curTriggerValue).$(curColumnPrefix)order_id
+----
+
 ====
 endif::pro[]
 
 ifndef::pro[]
 .Sample Trigger With External Select SQL that returns STORE_ID based on the ORDER_ID captured in the trigger.
-[source, SQL]
 ====
+
+[source, SQL]
+----
 insert into SYM_TRIGGER
 	(trigger_id,source_table_name,channel_id,external_select, last_update_time,create_time) 
 values ('orderlineitem', 'orderlineitem','orderlineitem',
 	'select STORE_ID from order where order_id=$(curTriggerValue).$(curColumnPrefix)order_id',
 	current_timestamp, current_timestamp);
+----
+
 ====
 endif::pro[]
 

symmetric-assemble/src/asciidoc/developer.ad

@@ -24,7 +24,7 @@ The INodeGroupExtensionPoint interface may be optionally implemented to indicate
  }
 ----
 
-===== IParameterFilter
+==== IParameterFilter
 Parameter values can be specified in code using a parameter filter. Note that there can be only one parameter filter per engine instance. The IParameterFilter replaces the deprecated IRuntimeConfig from prior releases.
 
 [source, java]
@@ -55,7 +55,7 @@ public class MyParameterFilter
 }
 ----
 
-===== IDatabaseWriterFilter
+==== IDatabaseWriterFilter
 Data can be filtered or manipulated before it is loaded into the target database. A filter can change the data in a column, save it somewhere else or do something else with the data entirely. It can also specify by the return value of the function call that the data loader should continue on and load the data (by returning true) or ignore it (by returning false). One possible use of the filter, for example, might be to route credit card data to a secure database and blank it out as it loads into a less-restricted reporting database.
 
 A DataContext is passed to each of the callback methods. A new context is created for each synchronization. The context provides a mechanism to share data during the load of a batch between different rows of data that are committed in a single database transaction.
@@ -99,55 +99,55 @@ The filter class should be specified in conf/symmetric-extensions.xml as follows
 </beans>
 ----
 
-===== IDatabaseWriterErrorHandler
+==== IDatabaseWriterErrorHandler
 Implement this extension point to override how errors are handled. You can use this extension point to ignore rows that produce foreign key errors.
 
-===== IDataLoaderFactory
+==== IDataLoaderFactory
 Implement this extension point to provide a different implementation of the org.jumpmind.symmetric.io.data.IDataWriter that is used by the SymmetricDS data loader. Data loaders are configured for a channel. After this extension point is registered it can be activated for a CHANNEL by indicating the data loader name in the data_loader_type column.
 
 SymmetricDS has two out of the box extensions of IDataLoaderFactory already implemented in its PostgresBulkDataLoaderFactory and OracleBulkDataLoaderFactory classes. These extension points implement bulk data loading capabilities for Oracle, Postgres and Greenplum dialects. See Appendix C. Database Notes for details.
 
 Another possible use of this extension point is to route data to a NOSQL data sink.
 
-===== IAcknowledgeEventListener
+==== IAcknowledgeEventListener
 Implement this extension point to receive callback events when a batch is acknowledged. The callback for this listener happens at the point of extraction.
 
-===== IReloadListener
+==== IReloadListener
 Implement this extension point to listen in and take action before or after a reload is requested for a Node. The callback for this listener happens at the point of extraction.
 
-===== IReloadVariableFilter
+==== IReloadVariableFilter
 Implement this extension point to filter the SQL used by the initial load to query source tables and purge target tables.  The extension receives the SQL and can replace variable names with values, which allows for adding new variables.  The org.jumpmind.util.FormatUtils.replace() method can be used to find and replace variables.
 
-===== ISyncUrlExtension
+==== ISyncUrlExtension
 This extension point is used to select an appropriate URL based on the URI provided in the sync_url column of sym_node.
 
 To use this extension point configure the sync_url for a node with the protocol of ext://beanName. The beanName is the name you give the extension point in the extension xml file.
 
-===== IColumnTransform
+==== IColumnTransform
 This extension point allows custom column transformations to be created. There are a handful of out-of-the-box implementations. If any of these do not meet the column transformation needs of the application, then a custom transform can be created and registered. It can be activated by referencing the column transform's name transform_type column of TRANSFORM_COLUMN
 
-===== INodeIdCreator
+==== INodeIdCreator
 This extension point allows SymmetricDS users to implement their own algorithms for how node ids and passwords are generated or selected during the registration process. There may be only one node creator per SymmetricDS instance (Please note that the node creator extension has replaced the node generator extension).
 
-===== ITriggerCreationListener
+==== ITriggerCreationListener
 Implement this extension point to get status callbacks during trigger creation.
 
-===== IBatchAlgorithm
+==== IBatchAlgorithm
 Implement this extension point and set the name of the Spring bean on the batch_algorithm column of the Channel table to use. This extension point gives fine grained control over how a channel is batched.
 
-===== IDataRouter
+==== IDataRouter
 Implement this extension point and set the name of the Spring bean on the router_type column of the Router table to use. This extension point gives the ability to programmatically decide which nodes data should be routed to.
 
-===== IHeartbeatListener
+==== IHeartbeatListener
 Implement this extension point to get callbacks during the heartbeat job.
 
-===== IOfflineClientListener
+==== IOfflineClientListener
 Implement this extension point to get callbacks for offline events on client nodes.
 
-===== IOfflineServerListener
+==== IOfflineServerListener
 Implement this extension point to get callbacks for offline events detected on a server node during monitoring of client nodes.
 
-===== INodePasswordFilter
+==== INodePasswordFilter
 Implement this extension point to intercept the saving and rendering of the node password.
 
 === Embedding in Android

symmetric-assemble/src/asciidoc/manage.ad

@@ -14,8 +14,27 @@ endif::pro[]
 
 ==== Add Node
 
+Multiple nodes can be hosted in a single SymmetricDS instance.  SymmetricDS will start a node for each properties file it finds in the engines directory.
+
+ifndef::pro[]
+Additional nodes can be added to the same SymmetricDS instance that the master node is running in or they can be added to a different
+SymmetricDS instance.  Either way, you create additional nodes by creating an <<Node Properties File>> with the `registration.url` set to 
+the `sync.url` of the master node.  When the SymmetricDS instance is restarted the new node will attempt to register with the master node.
+
+For the new node to fully become a part of the synchronization network <<Registration>> must be opened.
+endif::pro[]
+
+ifdef::pro[]
+
+Add nodes to your synchronization scenario by using the Node Setup Wizard.  This wizard is automatically shown when less the 2 nodes are configured.  
+It can also be accessed from the Manage Nodes screen by clicking the *Add* button. 
+
+image::wizard-connect/wizard-connect-menu.png[]
+
 include::manage/node-add.ad[]
 
+endif::pro[]
+
 ==== Load Data
 
 include::manage/node-load.ad[]
@@ -30,6 +49,81 @@ include::manage/node-registration.ad[]
 
 ==== Initial Loads
 
+Loading data for 3.8 and above has been modified, see <<Load Data>>.
+
+ifndef::pro[]
+
+When a load is requested it will either set the `initial_load_enabled` or the `reverse_initial_load_enabled` flag on the 
+appropriate <<NODE_SECURITY>> row.
+
+
+When the <<Route Job>> runs next, it will create batches that represent the initial load.  Batches will be created on the reload 
+channel for each table that is defined by <<Table Triggers>> and linked by <<Table Routing>> in the direction that the load 
+was requested.  The default reload channel is the _"reload"_ channel.  At the
+same time reload batches are inserted, all previously pending batches
+for the node are marked as successfully sent.
+
+Each table defined by <<Table Triggers>> and linked by <<Table Routing>> is represented by a reload <<OUTGOING_BATCH>>.  The batches 
+are inserted in the defined order.  If the `initial_load_order` is the same then SymmetricDS tries to determine the order the 
+tables need to be loaded in automatically based on foreign key dependencies.  A negative value for `initial_load_order` in <<Table Routing>> will result
+_no_ reload batch being inserted.
+
+If there are cyclical constraints, then foreign keys might need to be
+turned off or the initial load will need to be manually configured based
+on knowledge of how the data is structured.
+
+A SQL statement is run against each table to get the data load
+that will be streamed to the target node. The selected data is filtered
+through the configured router for the table being loaded. If the data
+set is going to be large, then SQL criteria can optionally be provided in `initial_load_select` 
+to pare down the data that is selected out of the database.
+
+IMPORTANT: Note that if the parent node that a node is registering with is _not_ a registration server node 
+(as can happen when using <<REGISTRATION_REDIRECT>> or when using multiple tiers) the parent node's <<NODE_SECURITY>> 
+entry must exist at the parent node and have a non-null value for column `initial_load_time`.  Nodes can't be registered 
+to a non-registration-server node without this value being set one way or another (i.e., manually, or as a result
+of an initial load occurring at the parent node).           
+
+===== Partial Initial Loads
+
+An efficient way to select a subset of data from a table for an initial
+load is to provide an `initial_load_select` clause in <<Table Routing>>
+. This clause, if present, is applied as a `where` clause to the SQL used to 
+select the data to be loaded. The clause may use "t" as an alias for the table being loaded, if needed. The
+`$(externalId)` token can be used for subsetting the data in the where clause.
+
+In cases where routing is done using a feature like
+<<Subselect Router>>
+, an
+`initial_load_select`
+clause matching the subselect's criteria would be a more efficient
+approach. Some routers will check to see if the
+`initial_load_select`
+clause is provided, and they will
+_not_
+execute assuming that the more optimal path is using the
+`initial_load_select`
+statement.
+
+One example of the use of an initial load select would be if you wished
+to only load data created more recently than the start of year 2011.
+Say, for example, the column
+`created_time`
+contains the creation date. Your
+`initial_load_select`
+would read
+`created_time &gt; ts {'2011-01-01 00:00:00.0000'}`
+(using whatever timestamp format works for your database). This then
+gets applied as a
+`where`
+clause when selecting data from the table.
+
+IMPORTANT: When providing an
+            `initial_load_select`
+            be sure to test out the criteria against production data in a query
+            browser. Do an explain plan to make sure you are properly using indexes.
+endif::pro[]
+
 include::manage/node-initial-load.ad[]
 
 ==== Send
@@ -38,6 +132,44 @@ include::manage/node-send.ad[]
 
 === Jobs
 
+Most work done by SymmetricDS is initiated by jobs. Jobs are tasks that are started and scheduled by a job manager.  
+Jobs are enabled by the `start.<name>.job` parameter.  
+
+Most jobs are enabled by default.  The frequency at which a job 
+runs is controlled by one of two parameters:  
+
+* `job.<name>.period.time.ms`
+* `job.<name>.cron`
+
+If a valid cron property exists in the configuration, then it will be used to schedule the job. Otherwise, the job manager 
+will attempt to use the period.time.ms property.
+
+ifdef::pro[]
+From the manage jobs screen the frequency that a job will run for the *current node* in the cluster can be managed.  If you want to change
+the frequency a job runs across a group or for everyone, you modify the parameter in the <<Parameters>> screen.  
+
+A job can also be _stopped_ and _restarted_.  If a job is scheduled to run infrequently and the job is needed to run immediately 
+you can select job and click *Run Now*.
+
+image::manage/manage-jobs.png[]
+endif::pro[]
+
+ifndef::pro[]
+
+The frequency of jobs can be configured in either the <<Node Properties File>> or in the <<PARAMETER>> table.  When managed in
+<<PARAMETER>> table the frequency properties can be changed in the master node and when the updated settings sync to the other nodes 
+in the system the job manager will restart the jobs at the new frequency settings.
+
+endif::pro[]
+
+SymmetricDS utilizes Spring's CRON support, which includes seconds as the first parameter. This differs from the typical Unix-based
+implementation, where the first parameter is usually minutes. For example,
+`*/15 * * * * *`
+means every 15 seconds, not every 15 minutes. See
+http://static.springsource.org/spring/docs/3.0.x/javadoc-api/org/springframework/scheduling/support/CronSequenceGenerator.html"[Spring's
+documentation]
+for more details.
+
 include::manage/jobs.ad[]
 
 === Installed Triggers
@@ -58,10 +190,53 @@ endif::pro[]
 
 === Outgoing Batches
 
+ifdef::pro[]
+
+The Outgoing Batches screens allows you to monitor data that has been <<_route_job,routed>>.
+
+image::manage/manage-outgoing-batches.png[]
+
+endif::pro[]
+
+Outgoing batches are delivered to the target node when the source node <<_push_job,pushes>> or when the target node <<_pull_job,pulls>>.
+
+A single push or pull connection is called a _synchronization_.
+
+For one synchronization, each enabled channel will be processed.  Channels are processed in the order defined by the 
+<<configuration/channels.ad#processing-order,Processing Order>> 
+setting on the <<_channels,channel>> with two exceptions:
+
+* If there are reload channels available to be sent _and_ the reload channels are _not_ in error, then only reload channels will be sent
+* If a channel is in error it will be moved to the bottom of the list 
+
+When outgoing batches are selected for a node and a channel, the maximum number of batches that are extracted per synchronization is 
+controlled by the <<configuration/channels.ad#max-batch-to-send,Max Batch To Send>> setting on the <<_channels,channel>>.
+
+There is also a setting that controls the max number of bytes to send in one synchronization. If SymmetricDS has extracted more than
+the number of bytes configured by the `transport.max.bytes.to.sync` parameter, then it will finish extracting the current batch and then finish
+synchronization so the client has a chance to process and acknowledge the "big" batch. This may happen before the configured 
+<<configuration/channels.ad#max-batch-to-send,Max Batch To Send>> has been reached.
+
+When extracting a batch, data is first extracted to the <<Staging Area>> and then sent across the network from the <<Staging Area>>.  
+The staging area is used to minimize the amount of time a database connection is being used when streaming
+over slower networks.  The use of the staging area can be turned off by setting the `stream.to.file.enabled` parameter.
+
 include::manage/outgoing-batches.ad[]
 
 === Incoming Batches
 
+ifdef::pro[]
+
+The Incoming Batches screens allows you to monitor batches that have been delivered to the target node.
+
+image::manage/manage-incoming-batches.png[]
+
+endif::pro[]
+
+Incoming batches are delivered to the target node when the source node <<_push_job,pushes>> or when the target node <<_pull_job,pulls>>.
+
+Incoming batches are written to the <<Staging Area>> first and then loaded.  The use of the staging area can be turned off by setting the `stream.to.file.enabled` parameter.
+
 include::manage/incoming-batches.ad[]
 
 include::manage/staging.ad[]

symmetric-assemble/src/asciidoc/manage/incoming-batches.ad

@@ -1,14 +1,3 @@
-ifdef::pro[]
-
-The Incoming Batches screens allows you to monitor batches that have been delivered to the target node.
-
-image::manage/manage-incoming-batches.png[]
-
-endif::pro[]
-
-Incoming batches are delivered to the target node when the source node <<_push_job,pushes>> or when the target node <<_pull_job,pulls>>.
-
-Incoming batches are written to the <<Staging Area>> first and then loaded.  The use of the staging area can be turned off by setting the `stream.to.file.enabled` parameter.
 
 ==== Incoming Sync Status