Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
0003160 - Add Utilities Documentation.
  • Loading branch information
jswilmer committed Jul 31, 2017
1 parent c468f82 commit f844c8e
Show file tree
Hide file tree
Showing 17 changed files with 351 additions and 2 deletions.
11 changes: 9 additions & 2 deletions symmetric-assemble/src/asciidoc/appendix.ad
Expand Up @@ -135,7 +135,7 @@ For major releases and clean upgrades, copy old settings into a new installation
. Copy old files from "security" folder. (These files contain keys for encryption.)
. Restart SymmetricDS.

=== Incremental Upgrade
=== Incremental Upgrade

ifdef::pro[]
The incremental upgrade method is for the open source version; SymmetricDS Pro users should use the <<Pro Upgrade>> method.
Expand All @@ -150,4 +150,11 @@ For patch and minor releases, copy and replace the library folders of the instal
. Restart SymmetricDS.

Most patch releases only change the JAR files named with "symmetric" in the "web/WEB-INF/lib" folder.


[appendix]
== Utilities

include::appendix/dbcompare.ad[]
include::appendix/dbimport.ad[]
include::appendix/dbexport.ad[]
include::appendix/dbfill.ad[]
58 changes: 58 additions & 0 deletions symmetric-assemble/src/asciidoc/appendix/dbcompare.ad
@@ -0,0 +1,58 @@
=== DBCompare

The DbCompare utility provides the ability to compare two SQL-based data sources and output a report of the differences, and optionally generate SQL to bring the target into sync with the source.

ifdef::pro[]
The dbcompare wizard is accessible from the explore tab of the pro console.
The pop-up window asks for source and target engines for the comparison as well as the the tables to be compared.

image::appendix/dbcompare/dbcompare-tab.png[width=600]

The next window allows specific tables to be selected for comparison.

image::appendix/dbcompare/dbcompare-table-select.png[width=600]

The results of the compare can be presented in four ways:

. Show Report
. Write Differences to SQL Editor
. Write Differences to File
. Apply Differences to Target Tables

The *Show Report* option will open a new tab in the SQL editor showing the results of the compare.

image::appendix/dbcompare/dbcomapre-results-table.png[]

The *Write Differences to SQL Editor* option will write all the required SQL to get the tables in sync and copy it to the current instance of the SQL editor.

image::appendix/dbcompare/dbcompare-write-sql.png[]

The *Write Differences to File* option will write the required SQL to get the tables in sync to the file system.

The *Write Differences to Target Tables* option will automatically execute the SQL necessary to bring the tables into sync.

endif::pro[]

ifndef::pro[]

The dbcompare utility can be run directly from the command line.
To run dbcompare enter the command *dbcompare [tablename...]* followed by any parameters from the command line in the bin directory of the Symmetric instance.
The *--help* parameter will display all the possible configurable parameters for the compare utility.
They are as follows:

- *--config <arg>* : A reference to a properties file path containing additional configuration for dbcompare. This properties file supports table where clauses in the format [table].[source|target].whereclause. A property called "where_clause" will specify a where clause used on all tables. For example, with a table called item, use item.where_clause=field > now()-2.
- *--exclude <arg>* : A comma-separated list of table names to exclude from comparison.
- *-h, --help* : Print help with each option and description.
- *--numeric-scale <arg>* : When comparing decimals, how many decimal places to consider while doing the comparison. Remaining digits will be rounded. Default value is 3.
- *--output-sql <arg>* : An output file for SQL statements that if executed on the target, should bring it into sync with the source.
- *-s, --source* : The source database engine properties file for comparison.
- *-t, --target <arg>* : The target database engine properties file for comparison.
- *--use-sym-config <arg>* : true|false. If true, sym_trigger, sym_transform, etc. will be consulted to build up the data model to compare. Default is true.

An example DbCompare results table is shown below:

image::appendix/dbcompare/dbcompare-results.png[width=1000]

The results table gives info from the compare as table columns. The data includes the number of rows *Matched*, *Different*, *Missing*, and *Extra* between the two source and target tables.

endif::pro[]
56 changes: 56 additions & 0 deletions symmetric-assemble/src/asciidoc/appendix/dbexport.ad
@@ -0,0 +1,56 @@
=== DBExport

The DbExport utility has the ability to export the structure of data from database tables to file.

ifdef::pro[]
The DbExport wizard is available from the explore tab of the pro console.
The popup window asks for the catalog, schema, and specific table(s) to export to a file.

image::appendix/dbexport/dbexport-database-export-tab.png[width=600]

The next tab allows for specific choices for the format and content of the exported file.

image::appendix/dbexport/dbexport-format-screen.png[width=600]

The format option allows for choices between *SQL*, *XML*, *CSV*, and *SYM_XML*.
The Compatibility tab allows for a choice between any of the supported SQL dialects.
The list of check boxes allows for more specific options for the content of the file including insert, drop or create scripts. A where clause can also be added to make the SQL more specific.

The last option allows for the scripts to be exported as either a file or directly to the SQL editor within the pro console.

endif::pro[]

ifndef::pro[]

The dbexport utility can be run directly from the command line.
To run dbexport enter the command *dbexport [tablename...]* followed by any parameters from the command line in the bin directory of the Symmetric installation.
The *--help* parameter will display all the possible parameters for the export utility.
They are as follows:

- *--add-drop-table* : Add drop table commands to output.
- *--catalog <arg>* : Look for tables in catalog
- *--compatible <arg>* : Change export to be compatible with a given database: db2, derby, firebird, greenplum, h2, hsqldb, hsqldb2, informix, interbase, mssql, mysql, oracle, postgres, sybase.
- *--debug* : Print debug information in logging.
- *--dir <arg>* : Indicate a directory to use for the export of files. If the directory is specified, then a file per table will be written to the directory.
- *-e, --engine <arg>* : The name of a configured engine. The name should correspond to an engine.name setting in one of the properties files in the engines directory.
- *--format <arg>* : Output format: SQL, CSV, XML, or SYM_XML.
- *-h, --help* : Print help with each option and description.
- *-i, --comments* : Write informational comments.
- *--no-create-info* : Do not write statements to create tables.
- *--no-data* : Do not write statements to insert into tables.
- *--no-foreign-keys* : Do not write statements to create foreign keys.
- *--no-indices* : Do not write statements to create indices.
- *--no-log-console* : No output will be sent to the console.
- *--no-log-file* : No output will be sent to the log file.
- *--no-qualifiers* : Do not qualify table and columns names with quoted identifiers.
- *-p, --properties <arg>* : The properties file with settings for the SymmetricDs engine. If not provided, defaults are used, then overridden with symmetric.properties values in user.home directory.
- *--schema <arg>* : Look for tables in schema.
- *--sql <arg>* : Run custom SQL to select data.
- *--use-jdbc-timestamp-format <arg>* : When true db export uses {ts 'yyyy-MM-dd h:mm:ss.SSS'} format for exported dates and times. Defaults to true.
- *--use-variable-dates* : Substitute dates as ${curdate+-millis}.
- *-v, --verbose* : Use verbose format for console output.
- *--where <arg>* : Specify the SQL for the "where" clause.

The same file formats utilized by dbimport are used for dbexport as well. Please see <<dbimport>> for format examples.

endif::pro[]
52 changes: 52 additions & 0 deletions symmetric-assemble/src/asciidoc/appendix/dbfill.ad
@@ -0,0 +1,52 @@
=== DBFill

The DbFill utility provides the ability to generate random data to populate any tables in Symmetric.

ifdef::pro[]

DBFill allows for any of a nodes tables to be filled with random tables. The popup window is accessible from the explore tab of the pro console.

image::appendix/dbfill/dbfill-tab.png[width=600]

The window asks for the Catalog, Schema, and selection of tables to fill.

image::appendix/dbfill/dbfill-settings.png[width=600]

The next window allows for a count of rows as well as the interval between execution. Insert, update, and, delete statements can also be specified along with options to ignore errors, and fill dependent tables.

endif::pro[]

ifndef::pro[]

The DbFill utility can be run directly from the command line.
To run dbfill enter the command *dbfill [tablename...]* followed by any parameters from the command line in the bin directory of the Symmetric installation.
The *--help* parameter will display all the possible parameters for the file utility.
They are as follows:

- *--cascade* : Include foreign key dependent tables not included in the table list.
- *--catalog <arg>* : Look for tables in catalog.
- *--commit <arg>* : number of rows to commit as transaction.
- *--commit-delay <arg>* : The time to wait in milliseconds after changing data before committing. The default is 0.
- *--continue* : Ignore ANY errors and continue to modify the database.
- *--count <arg>* : The number of rows to generate in each table.
- *--debug* : Print debug information in logging.
- *-e, engine <arg>* : The name of a configured engine. The name should correspond to an engine.name setting in one of the properties files in the engines directory.
- *-h, --help* : Print help with each option and description.
- *--ignore <arg>* : One or more prefixes to identify tables to ignore. This argument only works when no table names are provided. (e.g., "sym_,sys_")
- *--interval <arg>* : The time to wait in milliseconds between each transaction in the database.
- *--no-log-console* : No output will be sent to the console.
- *--no-log-file* : No output will be sent to the log file.
- *-p, --properties <arg>* : The properties file with settings for the SymmetricDS engine. If not provided. defaults are used, then overridden with first symmetric.properties in classpath, then overridden with symmetric.properties values in user.home directory.
- *--prefixed <arg>* : Prefix of table names to include.
- *--print* : Print out the SQL of the DbFill instead of filling the tables(s).
- *--rand* : Randomize number of rows to generate and commit.
- *--repeat <arg>* : Number of times to repeat the count of rows.
- *--rollback <arg>* : Percent chance (0-100) to perform a rollback.
- *--schema <arg>* : Look for tables in schema.
- *--select* : Select foreign key dependent data to satisfy constrains.
- *-v, --verbose* : Use verbose format for console output.
- *--weights <arg>* : By default, an insert is performed for each count ('1, 0, 0'). To randomly select between an insert, update or delete on each table, weight can be applied so inserts an occur more than deletes. To make sure inserts happen twice as much as updates, and deletes are never performed use '2,1,0'.

If the *print* parameter is given, by default, the insert statements for the fill will be sent to the command line.

endif::pro[]
176 changes: 176 additions & 0 deletions symmetric-assemble/src/asciidoc/appendix/dbimport.ad
@@ -0,0 +1,176 @@
[id="dbimport",reftext="DbImport"]
=== DBImport

The DbImport utility has the ability to import the structure and data of a tables(s) from a file to database tables.

ifdef::pro[]
The DbImport wizard is available from the explore tab of the pro console.
The popup window asks for the file format, catalog, schema, and tables to import to the database.
It also asks for the number of rows to commit, as well as multiple more specific options for the import.

image::appendix/dbimport/dbimport-tab.png[width=600]

Input files can be in SQL, XML, CSV, or SYM_XML file formats.

The following is an example of the *SQL* format used for DbImport.

[source,sql]
----
CREATE TABLE ITEM(
ITEM_ID INTEGER NOT NULL,
NAME VARCHAR(100) NULL,
PRIMARY KEY (ITEM_ID)
);
insert into ITEM (ITEM_ID, NAME) values (1805,'lVlmyxlADKEMZFIfWiipfiBqoeMHsSjsbgsoqBuEAWKhmJfaNA');
insert into ITEM (ITEM_ID, NAME) values (2649,'qjyWrDHXnsfdSDBqUzHMJPkRFQmwmWPWxBPPwaQmgpoQrqLcQC');
insert into ITEM (ITEM_ID, NAME) values (3334,'sCTJrMPFLauMvNrmYVyuLwcrinAVDVNmuHLULCWGYYmHRSmbsc');
insert into ITEM (ITEM_ID, NAME) values (3613,'xCNfevpgVBfegbrXHQOsChxCYPcwbjkpHXZpLFkLxMACaFWJnF');
insert into ITEM (ITEM_ID, NAME) values (3622,'UPTWPHujLPjFvusxAqzKSZCSFXXyhqfkbmholwDvuNdhLfIBGY');
----

The following is an example of the *CSV* format used for DbImport.

[source, csv]
----
"ITEM_ID","NAME"
"12","zSYwAyLGsbvsLhYFLBqmeprkfISVTlRnfHwsHFZcmZUpKQMXkT"
"174","WHQwyMhrWVZzmXyzzHiXRSyzfuOSImgqflupGrtLwmaGrOvFek"
"213","UlGKwTSLeygaxzkdTQxnvrSQMulBKbljpAkSVmbQCUONULUAlF"
"239","OOvqpLHIVcbOlJCXemNjCajcEJaBrPlneDkCZXqGYMXIuGRORq"
"471","RFHEaWcFtPDOkqEtbomEomuEWDQoAuyvJVnjPEsPpaqLxEuWpj"
----

The following is an example of the *XML* format used for DbImport.

[source,xml]
----
<database xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="dbimport">
<table name="ITEM">
<column name="ITEM_ID" primaryKey="true" required="true" type="INTEGER" size="10">
<platform-column name="h2" type="INTEGER" size="10"/>
</column>
<column name="NAME" type="VARCHAR" size="100">
<platform-column name="h2" type="VARCHAR" size="100"/>
</column>
</table>
<table_data name="ITEM">
<row>
<field name="ITEM_ID">1805</field>
<field name="NAME">lVlmyxlADKEMZFIfWiipfiBqoeMHsSjsbgsoqBuEAWKhmJfaNA</field>
</row>
<row>
<field name="ITEM_ID">2649</field>
<field name="NAME">qjyWrDHXnsfdSDBqUzHMJPkRFQmwmWPWxBPPwaQmgpoQrqLcQC</field>
</row>
<row>
<field name="ITEM_ID">3334</field>
<field name="NAME">sCTJrMPFLauMvNrmYVyuLwcrinAVDVNmuHLULCWGYYmHRSmbsc</field>
</row>
<row>
<field name="ITEM_ID">3613</field>
<field name="NAME">xCNfevpgVBfegbrXHQOsChxCYPcwbjkpHXZpLFkLxMACaFWJnF</field>
</row>
<row>
<field name="ITEM_ID">3622</field>
<field name="NAME">UPTWPHujLPjFvusxAqzKSZCSFXXyhqfkbmholwDvuNdhLfIBGY</field>
</row>
<table_data>
----

endif::pro[]

ifndef::pro[]

The DbImport utility can be run directly from the command line.
To run DbImport enter the command *dbimport [filename...]* followed by any parameters from the command line in the bin directory of the Symmetric installation.
The *--help* parameter will display all the possible parameters for the import utility.
They are as follows:

- *--alter* : If the table already exists, attempt to alter it to match the import definition. This only applies for --format=XML.
- *--alter-case* : During the creation of tables alter the case to match the default case of the database. This only applies for --format=XML.
- *--catalog <arg>* : Look for tables in catalog.
- *--commit <arg>* : The number of rows to import before committing data. The default is 10000.
- *--debug* : Print debug information in logging.
- *--drop-if-exists* : If the table already exists, attempt to drop it before creating it. This only applies for --format=XML.
- *-e, --engine <arg>* : The name of a configured engine. The name should correspond to an engine.name setting in one of the properties files in the engines directory.
- *--filter-classes <arg>* : A comma separated list of Java classes that implement org.jumpmind.symmetric.io.data.writer.IDatabaseWriterFilter. These filters will be applied to the import.
- *--force* : Ignore ANY errors and attempt to continue on processing the import data.
- *--format <arg>* : Input format: SQL, CSV, XML, or SYM_XML.
- *-h, --help* : Print help with each option and description.
- *--ignore* : Indicates that conflicts with existing rows should be ignored.
- *--interval <arg>* : The number of milliseconds to wait between committing transaction.
- *--no-log-console* : No output will be sent to the console.
- *--no-log-file* : No output will be sent to the log file.
- *-p, --properties <arg>* : The properties file with settings for the SymmetricDS engine. If not provided. defaults are used, then overridden with first symmetric.properties in classpath, then overridden with symmetric.properties values in user.home directory.
- *--replace* : Indicates that existing rows should be replaced. They will be replaced by an update statement. This only applies for --format=CSV, SYM_XML.
- *--schema <arg>* : Look for tables in schema.
- *--table <arg>* : Specify table to import.
- *--use-variable-dates* : Allow date subscription using format ${curdate+-millis}.
- *-v, --verbose* : Use verbose format for console output.

The following is an example of the *SQL* format used for DbImport.

[source,sql]
----
CREATE TABLE ITEM(
ITEM_ID INTEGER NOT NULL,
NAME VARCHAR(100) NULL,
PRIMARY KEY (ITEM_ID)
);
insert into ITEM (ITEM_ID, NAME) values (1805,'lVlmyxlADKEMZFIfWiipfiBqoeMHsSjsbgsoqBuEAWKhmJfaNA');
insert into ITEM (ITEM_ID, NAME) values (2649,'qjyWrDHXnsfdSDBqUzHMJPkRFQmwmWPWxBPPwaQmgpoQrqLcQC');
insert into ITEM (ITEM_ID, NAME) values (3334,'sCTJrMPFLauMvNrmYVyuLwcrinAVDVNmuHLULCWGYYmHRSmbsc');
insert into ITEM (ITEM_ID, NAME) values (3613,'xCNfevpgVBfegbrXHQOsChxCYPcwbjkpHXZpLFkLxMACaFWJnF');
insert into ITEM (ITEM_ID, NAME) values (3622,'UPTWPHujLPjFvusxAqzKSZCSFXXyhqfkbmholwDvuNdhLfIBGY');
----

The following is an example of the *CSV* format used for DbImport.

[source, csv]
----
"ITEM_ID","NAME"
"12","zSYwAyLGsbvsLhYFLBqmeprkfISVTlRnfHwsHFZcmZUpKQMXkT"
"174","WHQwyMhrWVZzmXyzzHiXRSyzfuOSImgqflupGrtLwmaGrOvFek"
"213","UlGKwTSLeygaxzkdTQxnvrSQMulBKbljpAkSVmbQCUONULUAlF"
"239","OOvqpLHIVcbOlJCXemNjCajcEJaBrPlneDkCZXqGYMXIuGRORq"
"471","RFHEaWcFtPDOkqEtbomEomuEWDQoAuyvJVnjPEsPpaqLxEuWpj"
----

The following is an example of the *XML* format used for DbImport.

[source,xml]
----
<database xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="dbimport">
<table name="ITEM">
<column name="ITEM_ID" primaryKey="true" required="true" type="INTEGER" size="10">
<platform-column name="h2" type="INTEGER" size="10"/>
</column>
<column name="NAME" type="VARCHAR" size="100">
<platform-column name="h2" type="VARCHAR" size="100"/>
</column>
</table>
<table_data name="ITEM">
<row>
<field name="ITEM_ID">1805</field>
<field name="NAME">lVlmyxlADKEMZFIfWiipfiBqoeMHsSjsbgsoqBuEAWKhmJfaNA</field>
</row>
<row>
<field name="ITEM_ID">2649</field>
<field name="NAME">qjyWrDHXnsfdSDBqUzHMJPkRFQmwmWPWxBPPwaQmgpoQrqLcQC</field>
</row>
<row>
<field name="ITEM_ID">3334</field>
<field name="NAME">sCTJrMPFLauMvNrmYVyuLwcrinAVDVNmuHLULCWGYYmHRSmbsc</field>
</row>
<row>
<field name="ITEM_ID">3613</field>
<field name="NAME">xCNfevpgVBfegbrXHQOsChxCYPcwbjkpHXZpLFkLxMACaFWJnF</field>
</row>
<row>
<field name="ITEM_ID">3622</field>
<field name="NAME">UPTWPHujLPjFvusxAqzKSZCSFXXyhqfkbmholwDvuNdhLfIBGY</field>
</row>
<table_data>
----

endif::pro[]
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit f844c8e

Please sign in to comment.