Merge branch '3.14' of https://github.com/JumpMind/symmetric-ds.git i…

…nto 3.14
JumpMind · May 9, 2022 · 5e1bb63 · 5e1bb63
2 parents 1df197b + aa7dadc
commit 5e1bb63
Show file tree

Hide file tree

Showing 71 changed files with 1,175 additions and 11,109 deletions.
diff --git a/README.md b/README.md
@@ -1,31 +1,34 @@
-<a href="https://sourceforge.net/projects/symmetricds/files/latest/download" rel="nofollow"><img alt="Download SymmetricDS" src="https://img.shields.io/sourceforge/dt/symmetricds.svg"></a>
-
-[![Build Status](https://dev.azure.com/jumpmind/symmetricds/_apis/build/status/SymmetricDS?branchName=3.10)](https://dev.azure.com/jumpmind/symmetricds/_build/latest?definitionId=4&branchName=3.10)
+<a href="https://sourceforge.net/projects/symmetricds/files/latest/download" rel="nofollow"><img alt="Download SymmetricDS" src="https://img.shields.io/sourceforge/dt/symmetricds.svg"></a> [![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)
 
 # SymmetricDS
-SymmetricDS is a database and file synchronization solution that is platform-independent, web-enabled, and database agnostic.  SymmetricDS was built to make data replication across two to tens of thousands of databases and file systems fast, easy and resilient.  We specialize in near real time, bi-directional data replication across large node networks over the WAN or LAN.
-
-SymmetricDS is sponsored by http://www.jumpmind.com.  JumpMind also provides support and a professional version that features a web-based user interface that simplifies configuration and management.
-
-You can find articles on SymmetricDS at http://www.jumpmind.com/blog and https://medium.com/data-weekly.
-
-Open source downloads here:
+SymmetricDS is database replication and file synchronization software that is platform independent, web enabled, and database agnostic. It is designed to make bi-directional data replication fast, easy, and resilient. It scales to a large number of nodes and works in near real-time across WAN and LAN networks. 
 
 <a href="https://sourceforge.net/projects/symmetricds/files/latest/download" rel="nofollow"><img alt="Download SymmetricDS" src="https://a.fsdn.com/con/app/sf-download-button"></a>
 
-Documentation is available at http://www.symmetricds.org/docs/overview.
+---
 
-Professional downloads are available at http://www.jumpmind.com/products/symmetricds/download.  Documentation is available at http://www.jumpmind.com/products/symmetricds/documentation.
+[Website](https://www.symmetricds.org) | [Documentation](https://www.symmetricds.org/docs/overview) | [Forum](https://sourceforge.net/p/symmetricds/discussion) | [Chat Room](https://gitter.im/symmetricds/Lobby) | [Issue Tracker](https://www.symmetricds.org/issues) | [Docker Hub](https://hub.docker.com/r/jumpmind/symmetricds)
 
-We also have developed a native SymmetricDS client.  Check out the code in the [symmetric-client-clib](symmetric-client-clib) and [symmetric-client-native](symmetric-client-native).
+---
 
-## Core Product Development
-We use Eclipse for development.
+## Developers
+Read the [Developer Overview](https://www.symmetricds.org/developer/overview) to get started.  You will need Java JDK 8 or later and Eclipse IDE.  To setup a development environment run the following commands:
 
-To setup a development environment run the following commands:
 ```
 cd symmetric-assemble
 ./gradlew develop
 ```
 
-This will generate Eclipse project artifacts.  You can then import the projects into your Eclipse workspace.
+This will generate Eclipse projects that can be imported into your workspace.  To contribute code changes back to the community, make sure you sign and submit the [Contributor License Agreement](https://www.symmetricds.org/developer/contributor).
+
+## Join the Community
+- Discuss on Forums: https://sourceforge.net/p/symmetricds/discussion
+- Chat on Gitter: https://gitter.im/symmetricds/Lobby
+- Q&A on Stack Overflow: https://stackoverflow.com/questions/tagged/symmetricds
+- Read JumpMind's Data Blog: https://www.jumpmind.com/blog
+- Read JumpMind's Data Weekly on Medium: https://medium.com/data-weekly
+- Follow on Facebook: https://www.facebook.com/symmetricds
+- Follow on Twitter: https://twitter.com/symmetricds
+
+## Sponsors
+SymmetricDS is sponsored by [JumpMind](https://www.jumpmind.com), who provides professional support, services, and software on top of the core open source software.  [SymmetricDS Pro](https://www.jumpmind.com/products/symmetricds/overview) is a professional version of the software that [adds enterprise features](https://www.jumpmind.com/products/symmetricds/editions), including a web user interface for configuration and management, support for clustering, bulk load performance, and enhanced security.  It's backwards compatible with SymmetricDS, so it's easy to [try out SymmetricDS Pro](https://www.jumpmind.com/products/symmetricds/try).
diff --git a/symmetric-assemble/Dockerfile b/symmetric-assemble/Dockerfile
@@ -1,17 +1,17 @@
-FROM openjdk:alpine
+FROM openjdk:8-jdk
+
+# Input the server zip file used to install SymmetricDS
+# This file must be in the Docker context (i.e. working directory)
+ARG SERVER_ZIP
 
 WORKDIR /app
+COPY $SERVER_ZIP /app/symmetric-ds.zip
 COPY README.md /app
 
-# Install editors for easy configuration
-RUN apk add --no-cache nano
-RUN apk add --no-cache vim
-
 # Install SymmetricDS
-RUN wget https://sourceforge.net/projects/symmetricds/files/latest/download -P /app -O symmetric-ds.zip
 RUN unzip /app/symmetric-ds.zip
 RUN rm -f /app/symmetric-ds.zip
-RUN mkdir /opt/
+RUN mkdir -p /opt/
 RUN mv symmetric-server* /opt/symmetric-ds
 RUN chmod -R u=rwX,g=,o= /opt/symmetric-ds/security
 

diff --git a/symmetric-assemble/Dockerfile.md b/symmetric-assemble/Dockerfile.md
@@ -0,0 +1,40 @@
+[![SymmetricDS](https://www.jumpmind.com/images/common/symmetricds.png)](https://www.jumpmind.com/products/symmetricds/overview)
+- - -
+This repository contains the JumpMind Inc. official Docker image for SymmetricDS. This Docker image is based on the openjdk:alpine image. This installation contains the default web server configuration for SymmetricDS.
+
+Overview
+===
+SymmetricDS is an open source database replication tool that is highly scalable and configurable.  SymmetricDS supports a wide variety of database platforms such as MySQL, Microsoft SQL Server, PostgreSQL, SQLite, Oracle SQL, and more.  Please visit the SymmetricDS website to learn more about the options and features: https://www.symmetricds.org/
+
+Running a SymmetricDS Container
+===
+To start SymmetricDS using HTTP run the following command:
+`docker run -p 31415:31415 --name sym jumpmind/symmetricds`
+
+***Please note that you must allow the IP of the Docker container to connect to the database in your database settings.  If you are running locally, allowing localhost is not sufficient since the Docker container is on a separate subnet.***
+
+Connecting to a Running Container
+===
+SymmetricDS may require manual configuration on the file system via command line tools.  To do this, run the following command to open a shell on a running container:
+`docker exec -it sym /bin/sh`
+
+This will open the default shell for Alpine Linux so that manual changes can be made on the container's file system.
+
+Volumes
+===
+Volumes allow data and files to be persisted across multiple containers.  This Docker image is configured to allow volumes for the engines, tmp, conf, and security directories so that configuration can be persisted. 
+
+To mount a volume, add one or more of the following argument to the run command:
+`-v sym-engines:/opt/symmetric-ds/engines`
+`-v sym-tmp:/opt/symmetric-ds/tmp`
+`-v sym-conf:/opt/symmetric-ds/conf`
+`-v sym-security:/opt/symmetric-ds/security`
+
+As an example, the following run command can be used to start SymmetricDS using HTTP and create the sym-engines, sym-conf, and sym-security volumes:
+`docker run -p 31415:31415 --name sym -v sym-engines:/opt/symmetric-ds/engines -v sym-conf:/opt/symmetric-ds/conf -v sym-security:/opt/symmetric-ds/security jumpmind/symmetricds`
+
+The above command will allow the engines, conf, and security directories to be persisted in the sym-engines, sym-conf, and sym-security volumes respectively.  If this container is stopped or deleted, a new container can be created using the same command and the configuration from the previous container will be retained.
+
+Building a SymmetricDS Image
+===
+`docker build -t symmetricds .`
diff --git a/symmetric-assemble/build.gradle b/symmetric-assemble/build.gradle
@@ -6,22 +6,25 @@ buildscript {
 
     dependencies {
         classpath 'org.asciidoctor:asciidoctorj-pdf:1.6.2'
-        classpath 'org.springframework.build.gradle:propdeps-plugin:0.0.7'
+        classpath 'org.asciidoctor:asciidoctor-gradle-jvm:3.3.2'
         classpath 'io.spring.gradle:docbook-reference-plugin:0.3.1'
 		classpath 'com.diffplug.spotless:spotless-plugin-gradle:6.4.2'
+		classpath 'com.bmuschko:gradle-docker-plugin:3.2.1'
+        classpath 'co.riiid:gradle-github-plugin:0.4.2'
     }
 }
 
 plugins {
     id "org.sonarqube" version "3.3"
 	id "com.github.hierynomus.license" version "0.15.0"
-	id 'com.vaadin' version '0.14.6.0' apply false
-	id 'org.asciidoctor.jvm.convert' version '3.3.2' apply false
+	id 'com.vaadin' version '0.14.8.1' apply false
 }
 
 apply plugin: 'java-library'
 apply plugin: 'idea'
 apply plugin: 'eclipse'
+apply plugin: 'com.bmuschko.docker-remote-api'
+apply plugin: 'co.riiid.gradle'
 
 allprojects {
 	apply plugin: 'com.diffplug.spotless'
@@ -48,6 +51,16 @@ task develop {
     dependsOn tasks.cleanEclipse, tasks.eclipse, tasks.cleanIdea, tasks.idea
 }
 
+github {
+    owner = 'JumpMind'
+    repo = 'symmetric-ds'
+    token = githubToken
+    tagName = version
+    targetCommitish = scmBranch.substring(scmBranch.lastIndexOf('/')+1)
+    name = version
+    prerelease = false
+}
+
 idea {
     project {
         ipr {
@@ -124,3 +137,40 @@ task releaseSymmetric {
 	description = 'Publish software to SourceForge website and online documentation to SymmetricDS website'
 }
 
+docker {
+    registryCredentials {
+        username = "$dockerUser".toString()
+        password = "$dockerPassword".toString()
+    }
+}
+
+task copyDockerReadme(type: Copy) {
+    from "Dockerfile.md"
+    into "../symmetric-server/build/distributions/README.md"
+}
+
+task copyDockerfile(type: Copy) {
+    dependsOn copyDockerReadme
+    from "Dockerfile"
+    into "../symmetric-server/build/distributions"
+}
+
+task buildImage(type: com.bmuschko.gradle.docker.tasks.image.DockerBuildImage) {
+    dependsOn copyDockerfile
+    inputDir = file("../symmetric-server/build/distributions")
+    tags = ["jumpmind/symmetricds:latest", "jumpmind/symmetricds:${version}".toString()]
+    buildArgs = ["SERVER_ZIP":"symmetric-server-${version}.zip"]
+}
+
+task publishDockerLatest(type: com.bmuschko.gradle.docker.tasks.image.DockerPushImage) {
+    dependsOn buildImage
+    imageName = "jumpmind/symmetricds:latest"
+}
+
+task publishDocker(type: com.bmuschko.gradle.docker.tasks.image.DockerPushImage) {
+    dependsOn publishDockerLatest
+    imageName = "jumpmind/symmetricds:${version}".toString()
+    doLast {
+        println "Pushing Docker Image to the jumpmind/symmetricds Repository"
+    }
+}
diff --git a/symmetric-assemble/common.gradle b/symmetric-assemble/common.gradle
@@ -45,6 +45,7 @@ allprojects {
                 artifact()
             }
         }
+
         mavenCentral()
         maven { url "https://maven.vaadin.com/vaadin-addons" }
         maven { url 'https://plugins.gradle.org/m2' }
@@ -85,17 +86,40 @@ subprojects { subproject ->
 
     apply plugin: 'com.github.hierynomus.license'
     apply plugin: 'java-library'
-    apply plugin: 'propdeps'
-    apply plugin: 'propdeps-maven'
-    apply plugin: 'propdeps-eclipse'
     apply plugin: 'maven'
     apply plugin: 'jacoco'
 
     configurations { deployerJars }
     configurations { testArtifacts.extendsFrom testRuntime }
 
-    configurations.provided.transitive = true
-    configurations.optional.transitive = false
+    configurations {
+    	provided {
+    		canBeResolved = true
+        	canBeConsumed = true
+        	visible = false
+        	transitive = true
+        	allDependencies.all {
+				dep -> configurations.default.exclude(group: dep.group, module: dep.name)
+			}
+        }
+        api.extendsFrom(provided)
+        project.conf2ScopeMappings.addMapping(300 + 1,
+			project.configurations.provided, "provided")
+		// Ensure only the compile configuration is exported
+		project.eclipse.classpath.file {
+			whenMerged { classpath ->
+				def resolved = project.configurations.compile.resolve()
+				classpath.entries.findAll{it.kind == 'lib'}.each {
+					it.exported = resolved.contains(project.file(it.path))
+				}
+			}
+		}
+		project.idea.module {
+			// IDEA internally deals with 4 scopes : COMPILE, TEST, PROVIDED, RUNTIME
+			// but only PROVIDED seems to be picked up
+			scopes.PROVIDED.plus += [project.configurations.provided]
+		}
+    }
 
     sourceCompatibility=1.8
     targetCompatibility=1.8

diff --git a/symmetric-assemble/gradle.properties b/symmetric-assemble/gradle.properties
@@ -1,6 +1,7 @@
 symAssembleDir=../symmetric-assemble
 version=3.14.x-SNAPSHOT
 scmVersion=?
+scmBranch=?
 deployUser=?
 deployPassword=?
 deployUploadUrl=?
@@ -11,7 +12,10 @@ publishPort=?
 publishServer=symmetricds.org
 sourceforgeUser=?
 sourceforgePassword=?
-vaadinVersion=14.7.2
+vaadinVersion=14.8.1
 org.gradle.daemon=false
 latestPreviousVersion=3.13.2
-restAPISecret=?
+restAPISecret=?
+dockerUser=?
+dockerPassword=?
+githubToken=?
diff --git a/symmetric-assemble/src/asciidoc/advanced-topics.ad b/symmetric-assemble/src/asciidoc/advanced-topics.ad
@@ -258,13 +258,13 @@ communication will be secured (typically the "root" or "central office" server).
 
 [source, cli]
 ----
-keytool -keystore keystore -delete -alias sym
+keytool -keystore keystore -delete -alias sym -storetype jceks
 ----
-IMPORTANT: If you receive a message like "keytool error: java.io.IOException: Invalid keystore format," try adding a parameter of "-storetype jceks". Or, if you receive a message like, "keytool error: java.lang.Exception: Alias <sym> does not exist" - then the alias does not exist and you can skip this step.
+IMPORTANT: If you receive a message like, "Alias <sym> does not exist" - then the key entry does not exist and you can skip this step.
 
 [source, cli]
 ----
-keytool -keystore cacerts -delete -alias sym
+keytool -keystore cacerts -delete -alias sym -storetype jks
 ----
 IMPORTANT: See above for possible errors from this command.
 [source]
@@ -276,7 +276,7 @@ Enter keystore password:  changeit
 
 [source, cli]
 ----
-keytool -keystore keystore -alias sym -genkey -keyalg RSA -validity 10950
+keytool -keystore keystore -alias sym -genkey -keyalg RSA -validity 10950 -storetype jceks
 ----
 
 [source]
@@ -306,14 +306,14 @@ Enter key password for <sym>
 
 [source, cli]
 ----
-keytool -keystore keystore -export -alias sym -rfc -file sym.cer
+keytool -keystore keystore -export -alias sym -rfc -file sym.cer -storetype jceks
 ----
 
 * Install the certificate in the trusted keystore.
 
 [source, cli]
 ----
-keytool -keystore cacerts -import -alias sym -file sym.cer
+keytool -keystore cacerts -import -alias sym -file sym.cer -storetype jks
 ----
 
 * Copy the cacerts file that is generated by this process to

diff --git a/symmetric-assemble/src/asciidoc/appendix/databases.ad b/symmetric-assemble/src/asciidoc/appendix/databases.ad
@@ -262,6 +262,16 @@ by database.
 |
 |
 
+|SingleStore
+|All
+|
+|✔
+|
+|✔
+|✔
+|
+|✔
+
 |SQL Anywhere
 |9
 |✔
@@ -486,6 +496,13 @@ When locating a table, SymmetricDS uses the default catalog and schema unless th
 |✔
 |select current_schema()
 
+|SingleStore
+| 
+|✔
+|select database()
+|
+|
+
 |SQL Anywhere
 |
 |✔
@@ -573,6 +590,7 @@ include::{includedir}/appendix/opensearch.ad[]
 include::{includedir}/appendix/oracle.ad[]
 include::{includedir}/appendix/postgresql.ad[]
 include::{includedir}/appendix/redshift.ad[]
+include::{includedir}/appendix/singlestore.ad[]
 include::{includedir}/appendix/snowflake.ad[]
 include::{includedir}/appendix/sqlite.ad[]
 include::{includedir}/appendix/sybase-ase.ad[]

diff --git a/symmetric-assemble/src/asciidoc/appendix/mssqlserver.ad b/symmetric-assemble/src/asciidoc/appendix/mssqlserver.ad
@@ -92,6 +92,11 @@ if it's not already enabled, which requires ALTER ANY DATABASE permission.
 [source, SQL]
 alter database mydb set change_tracking = on (change_retention = 2 days, auto_cleanup = on);
 
+Snapshot isolation must be enabled for the database to ensure all changes are included in replication.
+
+[source, SQL]
+alter database mydb set allow_snapshot_isolation on;
+
 For each table configured for replication, SymmetricDS will alter the table to enable change tracking,
 if it's not already enabled, which requires ALTER TABLE permission.
 

diff --git a/symmetric-assemble/src/asciidoc/appendix/singlestore.ad b/symmetric-assemble/src/asciidoc/appendix/singlestore.ad
@@ -0,0 +1,16 @@
+=== SinglStore
+
+SingleStore is only available in the professional version of SymmetricDS.  Supports bi-directional replication with SingleStore using log based data capture from the audit logs.  
+
+==== Requirements
+
+User provided to connect SymmetricDS to the SingleStore database must be different than the user account used by applications writing to SingleStore.  This is required for configurations where SingleStore is used as a source.
+
+Several database variables need to to be setup to support audit logging when SingleStore is a source node.   These variables can be set using the sdb-admin tool.
+
+.Setting variables
+----
+sdb-admin update-config --key auditlog_level --value ALL-QUERIES-PLAINTEXT --all
+sdb-admin update-config --key auditlog_disk_sync --value ON --all
+sdb-admin update-config --key auditlog_retention_period --value 7 --all
+----