Implementation of a lightweight CI/CD pipeline using git repositories for changes in COPPER workflows.
One of the basic ideas of COPPER is to be able to perform software changes in business workflows at runtime. With release 5.1 this now can be done by using git repositories.
copper2go makes use of this feature and should help to use COPPER in your project. All you need to do is, to start the copper2go container with your configuration, that support your business workflows.
Changes in the technical part of the application in addition to the business workflows can be deployed by the using traditional CI/CD pipelines.
In your container you can run workflows that are accessible via git. Just start with the ones in https://github.com/Keymaster65/copper2go-workflows.
-
Start container with
hello
andpricing
workflow-
docker run -d -p 19666:19666 --pull always --name copper2go --rm registry.hub.docker.com/keymaster65/copper2go:latest
-
-
In Browser
client
you can GET and see the used licenses -
In Browser
client
you can GET and see an exception-
http://localhost:19666/copper2go/3/api/twoway/2.0/Hello
will deliver a "IllegalArgumentException: A name must be specified."
-
-
Use any other HTTP
client
and POST your name to the URL-
Example:
curl --data Wolf http://localhost:19666/copper2go/3/api/twoway/2.0/Hello
-
Will produce someting like
Hello Wolf! Please transfer 4 cent
-
-
The following diagram shows the workflows used in the Demo.
-
The green
client
sends a HTTP request to the blue copper2gohello
workflow and waits for the response -
The blue copper2go
hello
workflow send a request the redpricing
backend and waits for the response -
The red
pricing
backend calculates a price send it in it’s response. To keep the demo simple, this backend also is a copper2go workflow.
You want to develop your own workflows? You may start with the existing ones.
-
Clone or fork the copper2go-workflows gradle project: https://github.com/Keymaster65/copper2go-workflows
-
Modify configuration and store it into environment variable C2G_CONFIG.
-
store it in your local docker host
config.json
-
Typically, modify workflowGitURI location
-
Start Container with your configuration:
-
docker run -d -p 19666:19666 -e C2G_CONFIG="$(cat config.json)" --pull always --name copper2go --rm registry.hub.docker.com/keymaster65/copper2go:latest
-
host.docker.internal
works for windows.
-
Start container with JMX on port 19665
-
docker run -d -e JAVA_TOOL_OPTIONS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=19665 -Dcom.sun.management.jmxremote.rmi.port=19665 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=false -Djava.rmi.server.hostname=host.docker.internal" -p 19665:19665 -p 19666:19666 --pull always --name copper2go --rm registry.hub.docker.com/keymaster65/copper2go:latest
-
Now you can visit the copper MBeas in tools like visualVM, JConsole etc.
-
Start the copper-monitoring Web Application on port 29665 using same JMX port
-
docker run -e JMX_HOST="host.docker.internal" -e JMX_PORT="19665" --name copperGui --rm -p 29665:8080 -d copperengine/copper-monitoring
-
Login with admin/admin at http://localhost:29665
-
Submit this setting and the TransientEngine should be visible
-
More Details can be found here https://github.com/copper-engine/copper-monitoring
COPPER was developed as an Orchestration Engine. For more than 10 years now, in 2021, many high performance systems are in production. The online configuration capability of this workflow engine is used seldom. To fill this gap, by using this feature as a main concept, copper2go was developed. By adding connectors, the development of Orchestration Services will become easier for Java developers.
You can run copper2go as a "Plattform as a Service", if you want to enable your clients to write and support their own COPPER workflows.
With an existing git repository, that contains all COPPER workflows of your orchestration services, copper2go is the enabler for "Software as a Service" orchestration systems.
Of course, copper2go containers can be run wherever you want. So the container may run
-
Internet Workflows, if hosted in the web
-
Intranet Workflows, if hosted in a company
-
Desktop Workflows, if run on your system
In times of automated build pipelines the needs for workflow systems are reduced, but a more lightweight git based pipeline might even better fit your needs. Here are some more advantages:
-
Lightweight pipeline form source code to deployment, because build is inside the copper2go container
-
Additional quality steps can be integrated into the git workflow
-
Using git and git workflows in the development teams
-
Reuse of copper2go images might reduce costs for images in the cloud
-
Unified copper2go images
-
Secure copper2go images
-
Easy extensions of copper2go images as forks on github
-
Easy extensions of copper2go connectors as forks on github
-
If the developers want to break the limits, they can use the vanilla-engines as forks on github
The copper2go application is checked using the https://plugins.gradle.org/plugin/org.owasp.dependencycheck, so engine does not contain any open known security issue, if a secure release of used 3rd party components are available. The Dependabot helps to keep the 3rd party components up to date (see https://github.com/dependabot). As workflows can not extend the used jars this check is sufficient for all workflow use cases.
High quality gates are define and used in sonarcloud. This is an important fact, if you want to stay on the secure side using copper2go. Version 5 of copper2go does not support secure connectors, but they can easily be added on demand.
Using git only, and compiling the workflow inside the container, you can very easy change your system’s behaviour "online". That is what want many people dream of, if they talk about "configuration". As one use case you can simply "revert" your changes, if something goes wrong. As the container is separated from the workflow, this "revert" always works by concept.
Non-blocking threads is one of the core concept of COPPER. That is the reason, why you can develop reactive high performance applications using COPPER or copper2go. A good motivation can be found in the "reactor" reference https://projectreactor.io/docs/core/release/reference/#intro-reactive
In spite of the non-blocking code with callback inside COPPER, the COPPER workflow Java code is readable and looks as simple as blocking code. There is no "callback hell" in your project. You might have a look at the "Motivation" of the Loom Project in https://cr.openjdk.java.net/~rpressler/loom/Loom-Proposal.html.
It is possible to start copper2go at CRaC (https://openjdk.org/projects/crac/) checkpoints. If you want to use it, you
must create a fitting pipeline to create images containing a checkpoint. To see how this can be done,
there are some files in the crac
directory.
The CRaC API is integrated in the HTTP connectors. Requirements to support Kafka and STDIN/STDOUT/STDERR can be found in the Backlog.
Last but not least, COPPER workflows can be executed for an unlimited time. It depends on the resources you add to the application. Transient workflows are supported in copper2go since release 0.1. Persistent workflows are supported by COPPER and currently in the Backlog of copper2go and will be added on demand.
copper2go bases on the COPPER (COmmon Persistable Process Excecution Runtime). To get more information about COPPER, you might visit https://github.com/copper-engine or https://github.com/copper-engine.
COPPER workflows look like Synchronous Java Code. This code is instrumented at compile time on the server. If you want to become more familiar with COPPER, that you might visit
The API described below is not much to read. I would suggest using the "Quick Start" and then start to develop a little sample application.
You came to a point, where you are missing a feature? No Problem. You might ues a pull request, if you want to add it by yourself. Of cource you can fork on GitHub. You may also add an item in https://github.com/Keymaster65/copper2go/issues
I am looking for feedback.
The Architecture overview
shows the main packages, classes and infaces of copper2go. Beside the workflows and the workflow api, you should be aware of the connector capabilities.
The copper2go Workflow’s API is hosted in the Maven Central. It can be found at several places
Starting with Workflow API 3.1.0 it also contains some dependencies, that extend the API.
Changes will be listed here in the "Released" chapter. In addition, you have the API to the COPPER framework.
Last but not least, the Java API can be used and is contained in the copper2go container.
Visit the sources, tests, examples and JavaDocs:
API | Link to JavaDoc |
---|---|
core |
|
extension |
|
extension |
|
JDK |
The configuration of the application and the receiver’s APIs will be listed here as Application API in the "Released" chapter.
No Java code except of Workflow’s API is released as an API. Of course, you can fork the project, if you want to make extensions.
The releases are hosted at github: https://github.com/Keymaster65/copper2go/releases
Docker images can be found here: https://hub.docker.com/r/keymaster65/copper2go
you find examples for the configuration of
-
COPPER Workflows (workflowRepositoryConfig)
-
HTTP Receiver (Server) (httpPort)
-
HTTP Request/Response (httpRequestChannelConfigs)
-
Kafka Server (kafkaHost, kafkaPort)
-
Kafka Receiver (kafkaReceiverConfigs)
-
Kafka Request/Response (kafkaRequestChannelConfigs)
The logback logging is defined in https://github.com/Keymaster65/copper2go/blob/master/copper2go-application/src/main/resources/logback.xml
There you can find the environment variables, that can be used to control logging at container start.
URLs path should be "/copper2go/3/api/TYPE/MAJOR.MINOR/WORKFLOW-NAME
where
-
The '3' relates to the HTTP Receiver API major version
-
The '2' Application API major version is still supported but DEPRECATED
-
TYPE can be
-
"twoway" if a body is expected in the reply. The HTTP reply is submitted during workflow processing.
-
"oneway if, no body is expected in the reply. The HTTP reply is submitted when workflow is initiated
-
-
MAJOR.MINOR is the version of the workflow
-
WORKFLOW-NAME is the target workflow of the request
URL "/" shows licence information.
If you want to add something, you may contribute with pull requests or forks. In a fork you might add 3rd party libs as wished.
Forks or Pull Requests are always very welcome.
Issues are very welcome, too.
-
The "master" branch is maintained and released as "latest" image on dockerhub.
-
The newest "release" branch is maintained and released as a tagged image for example "4.4.0"
-
Because build pipeline is completely automated after committing to "master" or "release", these branches are protected on github and commits must be signed "verified".
-
The newest Application API should be used in this document
-
For a "release", the following manual activities must be done
-
Move "Ongoing" block in "master"
-
Change "release" version in
.github/workflows/build.yml
in "master" on github (with a PR) -
merge "master" to "release" branch on github (with a PR)
-
"Draft a new release on github" on "release" branch with a copy of the moved "Ongoing" block (look at older releases for details)
-
-
Move "Ongoing" block in "master"
-
Update "version" in copper2go-api/build.gradle.kts
-
Push release branch
-
Publish artifact:
gradle :copper2go-api:publish
-
Login to https://oss.sonatype.org/
-
Find artifact in "Staging Repositories" and "Close" it and "Release" it
-
"Draft a new release on github" on "release" branch with a copy of the moved "Ongoing" block (look at older releases for details)
-
Update SNAPSHOT "version" in copper2go-api/build.gradle.kts
-
Push release branch
-
gradle dependencyUpdates
-
Deprecated:
gradle dependencies :sync-application:dependencies :vanilla-application:dependencies :application-framework:dependencies :copper2go-app:dependencies :scotty-engine:dependencies :sync-engine:dependencies :vanilla-engine:dependencies :copper2go-api:dependencies :connector-standardio:dependencies :connector-kafka-vertx:dependencies :connector-http-vertx:dependencies :connector-api:dependencies :engine-api:dependencies :pricing-simulator:dependencies --write-locks
-
Deprecated: Optional:
gradle dependencies :sync-application:dependencies :vanilla-application:dependencies :application-framework:dependencies :copper2go-app:dependencies :scotty-engine:dependencies :sync-engine:dependencies :vanilla-engine:dependencies :copper2go-api:dependencies :connector-standardio:dependencies :connector-kafka-vertx:dependencies :connector-http-vertx:dependencies :connector-api:dependencies :engine-api:dependencies :pricing-simulator:dependencies --write-locks --refresh-dependencies
-
gradle dependencyCheckAnalyze -x :dependencyCheckAnalyze --info
-
gradle clean build
-
gradle clean integrationTest
-
gradle -Pcopper2goVersion=tmp2 :copper2go-application:build :copper2go-application:jib
-
gradle systemTest
-
gradle :vanilla-application:build :vanilla-application:jib
-
gradle :sync-application:build :sync-application:jib
-
docker scout cves keymaster65/copper2go:latest
-
Optional:
gradle pitest -x :pitest
-
Optional:
gradle pitestReportAggregate
Of course, copper2go is ready use. Many more capabilities might be added. Here you find some of them ;-)
-
✓ Dependabot updates 04.06.2024
-
✓ Update crac to JDK 22
-
✓ Update to copper 5.5.2
-
✓ Dependabot updates 22.06.2024
-
✓ Use asciidoc instead of md
-
✓ Dependabot updates 21.07.2024
-
✓ More updates 21.07.2024
-
✓ Dependabot updates 17.08.2024
-
✓ Update to gradle 8.10
-
✓ Dependabot updates 13.09.2024
-
✓ Dependabot updates 04.10.2024
-
✓ Update to gradle 8.10.2
-
✓ Dependabot updates 19.10.2024
-
✓ Dependabot updates 27.10.2024
-
STDIN/OUT support in config and container (or remove it)
-
Load workflow subtree only from git
-
Operating
-
configure thread pool size, client pool size and more
-
Add some performance analysis
-
Collect Statistics and other (may be useful for Tests like Bridge-Test)
-
Remove version 2 of HTTP Receiver API
-
Extract WorkflowTestRunner.java and Copper2goDependencyInjector.java
-
Split copper2go-workflows; extract systemtest
-
Redesign DefaultRequestChannel (like Kafka). Use WARN instead of ERROR?
-
Redesign RequestChannel/EventChannel: Is the difference needed? Why 2 errorEvent (was inspired by STDOUT/ERR)?
-
Add test coverage for workflows to copper2go-workflows
-
Release internet workflow application as copper2go-webapp (see branch experiment/webapp)
-
Finish support kafka events
-
Add information "How Tos" to developer’s guide
-
Overview
-
Request Channel Stores
-
Event Channel Stores
-
Configuration Reply Channel Store
-
Tickets
-
Workflow Development/Test
-
Support of HTTP callbacks
-
Add callback in WorkflowData ("replychannel")
-
Use vanilla-engine without a github fork
-
Extend connectors without a github fork
-
Extend copper2go image without a github fork
-
Retire vertx
-
Replace vertx HTTP components with simpler implementation
-
Replace vertx Kafka components with simpler implementation
-
On demand only
-
HTTP Security
-
Kafka Security
-
CRaC Support for STDIN/STDOUT/STDERR
-
Support Binary data
-
Binary Binding
-
Kafka choreography example
-
CRaC Support for Kafka
-
PostgreSQL for workflow persistent instances
-
Async idempotent DB API
-
PostgreSQL support for business resources
-
JMS support (may be IBM MQ, ActiveMQ or …)
-
factoryfx integration
-
extend workflow attributes to a MultiMap
-
Withdrawn
-
Add new Workflow Repository for Performancetest
-
Delete .copper on start (if still problems occurs)
-
Workflow with XML binding (may be not ;-)
-
Vertx Bus Connector
-
✓ Update Application API to JDK 22
-
✓ Use a registered port in application
-
✓ Dependabot updates 02.12.2023
-
✓ Dependabot updates 10.12.2023
-
✓ Dependabot updates 15.12.2023
-
✓ Dependabot updates 23.12.2023
-
✓ Dependabot updates 27.12.2023
-
✓ Dependabot updates 06.01.2024
-
✓ Dependabot updates 12.01.2024
-
✓ Dependabot updates 20.01.2024
-
✓ Dependabot updates 05.02.2024
-
✓ Dependabot updates 24.02.2024
-
✓ Dependabot updates 02.03.2024
-
✓ Dependabot updates 24.03.2024
-
✓ Dependabot updates 26.03.2024
-
✓ Dependabot updates 29.03.2024
-
✓ Dependabot updates 30.03.2024
-
✓ No more gradle.lockfile usage
-
✓ Dependabot updates 14.04.2024
-
✓ Dependabot updates 20.04.2024
-
✓ Dependabot updates 20.05.2024
-
✓ Dependabot updates 23.05.2024
-
✓ Dependabot updates 02.06.2024
-
✓ Toolchain Update to JDK 22
-
✓ Update jackson-databind from 2.15.3 to 2.17.1
-
✓ Update slf4j-api from 2.0.10 to 2.0.13
-
✓ Update copper-coreengine from 5.4.2 to 5.5.0
-
✓ Workflow with Json binding
-
✓ Use org.crac:crac instead of io.github.crac:org-crac
-
✓ New ENV for logback $LOG_LEVEL_ROOT
-
✓ Dependabot updates 21.10.2023
-
✓ Dependabot updates 01.11.2023
-
✓ Dependabot updates 04.11.2023
-
✓ Dependabot updates 12.11.2023
-
✓ Dependabot updates 18.11.2023
-
✓ Build with gradle 8.5 and use JDK 21 for gradle
-
✓ Much faster gradle INIT on Windows with org.gradle.vfs.watch=true
-
✓ Separate OWASP github workflow
-
✓ Update copper-coreengine from 5.4.1 to 5.4.2
-
✓ Update slf4j-api from 2.0.6 to 2.0.9
-
✓ Update jackson-databind from 2.14.1 to 2.15.3
-
✓ CRaC integration in VertxHttpClient
-
✓ CRaC integration in Copper2GoGitWorkflowRepository
-
✓ Support faster startup using CRaC (https://openjdk.org/projects/crac/)
-
✓ Update netty-handler to 4.1.100.Final (Continue suppressing CVE-2023-4586)
-
✓ Fix race condition in WorkflowHandler for oneway requests
-
✓ Dependabot updates 14.10.2023
-
✓ Full automated build pipelines into dockerhub releases for "latest/master" and "release"
-
✓ CRaC integration in VertxHttpServer *
-
✓ JMX usage in Container
-
✓ Support of COPPER core GUI
-
✓ (Security) Updates
-
✓ Update to JDK 21
-
✓ Add pitest support (https://pitest.org/)
-
✓ Add a sync-engine and application with blocking code
-
✓ Run sync-application on JDK 19 with Loom enabled
-
✓ Security updates
-
✓ Additional updates
-
✓ Add licenses to sync-application
-
✓ Do not use root as user to run copper2go
-
✓ Add license info "vanilla" engine implementation
-
✓ Add "vanilla" engine implementation
-
✓ Add "vanilla" engine implementation as incubating feature
-
✓ Fix CVE-2022-38752 by excluding snakeyaml
-
✓ Security updates
-
✓ Version updates
-
✓ Make logLevel accessible via environment
-
✓ Add log configuration here and changed defaults
-
✓ Support for Java 17
-
✓ Add Bridge Workflow
-
✓ Add quality badges
-
✓ Support HTTP URL parameter (in but no out)
-
✓ Support Kafka Header (parameter) (in and out)
-
✓ Improve unit test coverage up to 95% or more
-
✓ Update many used jars
-
✓ Add use case in Developer’s Guide
-
✓ Refactor engine subproject for tests and extracting engine-api and connector-api
-
✓ Refactor connector subproject for tests and extracting more modules
-
✓ Fix memory leak in ReplyChannelStore
-
✓ UUID might not be set in WorkflowData (breaking change)
-
✓ Use term "oneway" instead of (incoming) "event" in code (breaking change)
-
✓ Use term "oneway" instead of (incoming) "event" in this README
URLs path should be "/copper2go/3/api/TYPE/MAJOR.MINOR/WORKFLOW-NAME
where
-
The '3' relates to the Application API major version
-
The '2' Application API major version is still supported but DEPRECATED
-
TYPE can be
-
"twoway" if a body is expected in the reply. The HTTP reply is submitted during workflow processing.
-
"oneway if, no body is expected in the reply. The HTTP reply is submitted when workflow is initiated
-
-
MAJOR.MINOR is the version of the workflow
-
WORKFLOW-NAME is the target workflow of the request
URL "/" shows licence information.
-
✓ Support HTTP URL parameter (in but no out)
-
✓ Support Kafka Header (parameter) (in and out)
-
✓ Remove payload member from WorkflowData (breaking change)
-
✓ Require Java 17 (breaking change)
-
✓ Restructure packages (breaking change)
-
✓ Kafka Connectors
-
✓ Separate System- and Integration- testing in Build Pipeline
-
✓ Add licence info more files and update year
-
✓ Add licence URL in non Docker apps
-
✓ Add API release info
-
✓ Correct URLs in Quick Start
-
✓ Replace the lax Application API by a stricter validation
-
publish to public artifact repository
-
Add How to develop Workflows for IDEA
-
Add How to develop Workflows for gradle
-
Draft development Guide
Now the API is available in maven. Due to the OSSRH process, the package were refactored. https://repo1.maven.org/maven2/io/github/keymaster65/copper2go-api/
URLs path must be "/copper2go/2/api/TYPE/MAJOR.MINOR/WORKFLOW-NAME
where
-
The '2' relates to the Application API major version
-
TYPE can be
-
"request" if a body is expected in the reply
-
"event" if, no body is expected in the reply
-
-
MAJOR.MINOR is the version of the workflow
-
WORKFLOW-NAME is the target workflow of the request
URL "/" shows licence information.
-
http server support
-
http client support
-
non-business config
-
Multi workflow
-
docker image distribution
-
add license info
-
Add Application and Workflow path
-
Event consuming workflows (no reply body, state 202 only)
-
quick start
The payload is now part of the WorkflowData https://github.com/Keymaster65/copper2go/tree/release/1.0/src/main/java/de/wolfsvl/copper2go/workflowapi
Support for configuration of the HTTP server and https://github.com/Keymaster65/copper2go/blob/release/1.0/src/main/resources/de/wolfsvl/copper2go/application/config/config.json
URLs must end with /MAJOR.MINOR/WORKFLOW-NAME
where
-
MAJOR.MINOR is the version of the workflow
-
WORKFLOW-NAME is the target workflow of the request
-
0.1 A first MVP (Minimum Viable Product)