Please note that this project is a work in progress.
CANCHITO-WORKFLOW-MANAGER (CWM) is a powerfull and yet light-weight and easy to use solution for handling workflows and business processes. At its core is a high performance open-source business process engine based on Flowable with the flexibility and scalability to handle a wide variety of critical processes.
We really hope that the additional features added to Flowable can be as beneficial as we hope they are. If you would like to have a look into our other projects, please visit us at Canchito-Dev.
- Extremely simple, easy to understand
- Simple but clean structure
- Easy to scale
- Front- and Back-end are decoupled, allowing you to have them running in individual servers
- Commented code
- Integrates with a range of enterprise clouds, including Amazon Web Service, Microsoft Azure and Google Drive
- Integrates with FFmpeg and MediaInfo
- Custome palette perfectly compatible with Flowable Designer plug-in for Eclipse
Help us find bugs, add new features or simply just feel free to use it. Download CANCHITO-WORKFLOW-MANAGER (CWM) from our GitHub site.
The source code can be found in our GitHub site.
back-end: has the source code for the back-endfront-end: has the source code for the fron-endpalette: contains the source code for the Eclipse palette
Here is a detail explanation of the back-end modules:
cwm-app: Main module. Has all the main resources and configurationscwm-cloud: Has all the implementations to use the different cloud servicescwm-core: Handles CWM's Async Job Executorcwm-dao: Handles all the database accesses. You can find all the repositories and entities herecwm-flowable: Implementations created for seamlessly integrate with Flowablecwm-rest-api: Rest controllers for CWMcwm-service: Handles the business logiccwm-cloud: Contains all the task behaviors and runnables used in CWM's Async Job Executorcwm-utils: Has all the app's utils
The MIT License (MIT)
Copyright (c) 2017, canchito-dev
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
This project started as a way of helping me understand and learn about Flowable and Spring Boot. But it grew up to include some other features that have been helpful to me. It might not perfectly coded, but it was developed with all the good intensions at heart.
This is a totally unpaid and voluntary project, developed during my free-time and shared on GitHub. It can be used for private or commercially, without a cost. There is only one simple request, do not bash or complain. If you don't like it, don't use it.
If you find an issue, please fill free and open a ticket, we will try to solve it as soon as possible. If you would like to become an active contributor to this project, even better.
If you would like to become an active contributor to this project please follow theses simple steps:
- Fork it
- Create your feature branch
- Commit your changes
- Push to the branch
- Create new Pull Request
- Decoupled front- and back-end, allowing them run in individual servers
- Integration with a range of enterprise clouds, including Amazon Web Service, Microsoft Azure and Google Drive
- Integration with MediaInfo
- Integration with FFmpeg
- Allow execution of long-running tasks
- Prioritize process instances
- File transferring by FTP and Samba
- High-available, redundant and scalable clustered platform
Flowable is a light-weight workflow and Business Process Management (BPM) Platform targeted at business people, developers and system admins. Its core is a super-fast and rock-solid BPMN 2 process engine for Java. It's open-source and distributed under the Apache license. Flowable runs in any Java application, on a server, on a cluster or in the cloud. It integrates perfectly with Spring, it is extremely lightweight and based on simple concepts.
In order to understand the main "limitation", you need to understand how Flowable's asyc job executor works. Please read the following section from Flowable's user guide:
And according to Frederik Heremans (one of Flowable's main developers), in his reply to this question rised in Flowable's official forum, we also know that when executing long-running tasks the async job executor, bahaves as follow:
- Executing a service-task (or any other task) keeps a transaction open until a wait-state/process-end/async-task is reached. If you have long-running operations, make sure your database doesn't time out
- When a jobs is running for 5 minutes, the job aquisistion-thread assumes the async job executor that was running the job, has either died or has failed. The lock of the job is removed and the job will be executed by another thread in the executor-pool. This timeout-setting can be raised, if that is required
- Long-running tasks modeled in the flowable-process always keep a transaction open and a async job executor thread occupied. Better practice is to use a queue-signal approach where the long-running operation is executed outside of Flowable (queued to eg. camel using a service-task, providing the neccesary variables needed alongside). When the long-running task is completed, it should signal the execution, which has a recieve-task modeled in
Due to the requirements needed on CANCHITO-WORKFLOW-MANAGER (CWM), the regular way Flowable's async job executer behavior was a "limitation". But they are not actual limitations. They are simply the way in which the engine should behave.
Following Frederik Heremans reply, with CANCHITO-WORKFLOW-MANAGER (CWM) it was decided to implement the queue-signal approach together with a database table, which will contain the tasks queue.
For the front-end, you need a standard Web server with at least:
- PHP 5.3.2 or higher
- Make sure mod_rewrite is enabled and activated
- Basic knowledge of Composer
For your development environment, you can use WAMPServer, which is a Windows web development environment, and stands for Windows, Apache, MySQL and PHP.
Composer is a very simple and easy to use dependency manager for PHP. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you. You can download the Windows installer from their official site.
Now, the back-end runs on a JDK higher than or equal to version 7. Go to Oracle Java SE site and download it. There are installation instructions on that page as well.
In order to make sure that the installacion was successful, run java -version on the command line. That should print the installed version of your JDK:
java -version
java version "1.8.0_141"
Java(TM) SE Runtime Environment (build 1.8.0_141-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.141-b15, mixed mode)
Finally, you need a MySQL database. If you used WAMPServer, it should automatically be installed.
A folder handler is a tag or an alternative name for referencing an actual directory in a FTP or Samba network location, a local storaga or even one in one of the supported clouds. They are used for describing actual paths and how to access them. Folder handlers simplify how you refer to directories, as you only specify a short and descriptive name, instead of a complex and longer path plus file name.
Moreover, folder handler' path and credentials only need to be updated once, and automatically, all actions using that folder will be updated.
A file handler is used as a tag for referencing a file within a workflow. Thanks to file handlers, you do not need to use the full path and name of each file in every step of a workflow. Consequently, it is only necessary to assign a file handler to a file, and CWM associates the full path and file name of the actual file during each step of the workflow.
It is required that each file is associated to a unique file handler. If the same file handler is used for a second file, the first values are overwritten.
Create the database schema and all the database tables. Simply execute the .sql statements in the database- folder (with PHPMyAdmin for example).
Copy the content of the front-end- folder into the Web server default folder and just follow these steps:
- Make sure you have mod_rewrite activated on your server / in your environment.
- Install Composer and run
composer installin the project's folder to download the dependencies and create the autoloading stuff from Composer automatically. For those who are not familiar with Composer, just remember back in the days, when you were using a PHP files with all the includes you needed. Well, Composer creates classes that automatically do this. - Create a virtual host. For instance
cwm.canchito-dev.com. - Go to folder
application/configand open the fileconfig.php. Here you will have to modify some parameters.TENANT_ID_PART: Configuration for specifying which part of the$_SERVER['HTTP_HOST']variable use as a tenant id. For instance:$_SERVER['HTTP_HOST'] = cwm.canchito-dev.com. The system splits 'nahual.domain' by the '.' into two substrings
If TENANT_ID_PART = 0, 'cwm' will be used as TENANT_ID
If TENANT_ID_PART = 1, 'canchito-dev' will be used as TENANT_ID and so onCWM_URL_DOMAIN: the domain where the back-end is runningCWM_URL_PORT: the port in which the back-end is listeningCWM_URL_SUB_FOLDER: the path of the main dispatcher servlet
Open the configuration file src/main/resources/application.properties and modify:
server.address: Network address to which the server should bind toserver.connection-timeout: Time in milliseconds that connectors will wait for another HTTP request before closing the connection. When not set, the connector's container-specific default will be used. Use a value of -1 to indicate no (i.e. infinite) timeoutserver.display-name: Display name of the applicationserver.servlet-path: Path of the main dispatcher servletserver.port: Server HTTP portserver.tomcat.max-connections: Maximum number of connections that the server will accept and process at any given timeserver.tomcat.max-threads: Maximum amount of worker threadsserver.tomcat.uri-encodin: Character encoding to use to decode the URI
For instance:
server.address=localhost
server.connection-timeout=60000
server.display-name=Canchito Workflow Manager
server.servlet-path=/canchito-dev-rest
server.port=10000
server.tomcat.max-connections=100
server.tomcat.max-threads=100
server.tomcat.uri-encoding=UTF-8
Mails can be sent from both the front- and the back-end. Let's first do the configuration for the front-ent by opening the application/config/config.php file and modifying the following parameter to suit your needs:
IS_SMTP: Tells PHPMailer to use SMTPMAIL_SERVER: Sets the hostname of the mail serverSMTP_PORT: Sets the SMTP port number - 587 for authenticated TLS, a.k.a. RFC4409 SMTP submissionSMTP_USERNAME: Username to use for SMTP authentication - use full email address for gmailSMTP_PWD: Password to use for SMTP authenticationSMTP_DEBUG: Enable SMTP debugging: (0) = off (for production use); (1) = client messages; (2) = client and server messages. Default is 0SMTP_SECURE: Set the encryption system to use - ssl (deprecated) or tls. Default is tlsSMTP_AUTH: Whether to use SMTP authentication. Default is trueMAIL_CHARSET: Sets the character setDEBUG_OUTPUT: Ask for HTML-friendly debug output. Default is html
Now, the mail configuration for the back-end is needed so that Flowable can send mails by its mail task. To configure it, open the file src/main/resources/application.properties and modify the following fields:
flowable.mail-server-default-from: The name of the sender as it will appear in mailflowable.mail-server-host: SMTP server host. For instancesmtp.example.comflowable.mail-server-password: Login password of the SMTP serverflowable.mail-server-port: SMTP server portflowable.mail-server-use-tls: Use tls or notflowable.mail-server-user-name: Login user of the SMTP server
At the moment, the system can integrate with Amazon Web Services (AWS), but the roadmap includes integration with other clouds such as Microsoft's Azure and Google. For any cloud, it is required a valid account. To obtain a free AWS account, go to the AWS Free Tier home page and click Create A Free Account/Create an AWS Account. Signing up will enable you to use all of the services offered by AWS.
To configure it, open the file src/main/resources/application.properties and modify the following fields:
amazon.credentials-access-key-id: The access key of the accountamazon.credentials-secret-access-key: The secret access key related to the accountamazon.configuration-region: The region to which the account should log in
The database configuration is done in the src/main/resources/application.properties file. Simply, modify the following parameter to suit your needs:
spring.datasource.continue-on-error: Do not stop if an error occurs while initializing the databasespring.datasource.driver-class-name: Fully qualified name of the JDBC driver. Auto-detected based on the URL by defaultspring.datasource.password: Login password of the databaserspring.datasource.url: JDBC url of the databasespring.datasource.username: Login username of the databasecwm.datasource.*: Datasource specific settings
For instance:
spring.datasource.continue-on-error=false
spring.datasource.url=jdbc:mysql://localhost:3306/database?autoReconnect=true
spring.datasource.username=username
spring.datasource.password=password
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
cwm.datasource.test-on-borrow=true
cwm.datasource.test-on-return=true
cwm.datasource.test-while-idle=true
cwm.datasource.validation-query=SELECT 1
cwm.datasource.validation-interval=30000
cwm.datasource.connection-timeout=60000
cwm.datasource.connection-test-query=SELECT 1
cwm.datasource.log-validation-errors=true
cwm.datasource.max-idle=1
cwm.datasource.min-idle=1
cwm.datasource.max-active=5
cwm.datasource.initial-size=5
cwm.datasource.time-between-eviction-runs-millis=5000
cwm.datasource.min-evictable-idle-time-millis=30000
Put in simple words, CANCHITO-WORKFLOW-MANAGER (CWM)'s async job executor are individual threads that are started once when the application is started. Each thread starts a thread pool that reuses a (configurable) fixed number of threads operating off database table called CWM_TASKS_QUEUE and acting as a priority blocking list, using the provided ThreadFactory to create new threads when needed. At any point, at most n threads will be active processing tasks.
Periodically, pending tasks are pulled from the database. The number of pending tasks that are pulled at once, depends on the number of available threads on CANCHITO-WORKFLOW-MANAGER (CWM)'s async job executor for a specific task type. If additional tasks are submitted when all threads are active, they will reside in the database until a thread is available.
If any thread terminates due to a failure during execution prior to shutdown, a new one will take its place if needed to execute subsequent tasks. The threads in the pool will exist until it is explicitly shutdown.
Once a task is pulled by a thread, it is locked. By locking it, we can have several CANCHITO-WORKFLOW-MANAGER (CWM) simultaneously running on different servers. Thus allowing us to have a fail-over, clustered system.
If two or more CANCHITO-WORKFLOW-MANAGER (CWM)'s async job executor are started, they all will be raising for pulling the pending tasks. Consequently, only the first one that looks the task, will be the one that will execute it. The other async job executor will get an optimistic locking exception.
In order to understand the way long-running tasks are added to the queue, lets have a look at a very simple workflow as the one in the below image. As you can see, it is composed of a start event, a copy task (which is a service task), and an end event.
The copy task is a long-running service task, which needs to be processed by the async job executor. Long-runing tasks in CANCHITO-WORKFLOW-MANAGER (CWM) extend AbstractTaskCanchitoBehavior, which at the same time extend from Flowable's TriggerableActivityBehavior class.
The TriggerableActivityBehavior is the parent class for many BPMN 2.0 task types such as ReceiveTask, ScriptTask, UserTask, etc. This class provides two methods: execute()and trigger(). The class AbstractTaskCanchitoBehavior provides a two methods: submitTask() and checkSignal(DelegateExecution execution). These four methods are the pillars for creating a long-running task implementing the TriggerableActivityBehavior instead of adding two BPMN task (send task and receive task) in your process diagram.
The execute(DelegateExecution execution) method is invoked when the service task is entered. In our case, it is typically used for data valiation and preparation. There is no business logic here. For instance, it validates that all the needed information for the task to be correctly executed has valid values.
The submitTask(DelegateExecution execution, JSONObject details, String beanId) method submits an asynchronous task to the actual service. The submit action is actually storing the task in the database table by calling the save() method from the GenericTaskService class. You can modify this method according to your database table structure.
After submitting the task and the method returns, the process engine will not continue execution. The TriggerableActivityBehavior acts as a wait state. This means, that the process instances is put on hold, until a trigger is received.
Periodically, the AcquireTaskThread in charge of these kind of service task (in this example, the copy task), read the task que database table. When it finds a new task, it acquires it, and locks it. Once locked, it calls the run() method. Here, you will find the business logic for the invoked service task. For our examples, it will problably call functions and method to copy a file from one location to another. When done, AcquireTaskThread will call the trigger() method.
The trigger(DelegateExecution execution, String signalName, Object signalData) method is invoked as the process engine is being triggered by the callback. The trigger() method is responsible for leaving the service task activity and allowing the normal flow of the process instance. But before leaving the service task, the checkSignal() is called. It is in this method, were the execution of the service task's logic is analyzed and determined if it finished correctly or with errors.
By having a separate thread pool for executing long-running tasks, CANCHITO-WORKFLOW-MANAGER (CWM) has decoupled the process engine from the service implementation. From the point of view of Flowable's process engine, the TriggerableActivityBehavior is a wait state: after the execute() method returns, the process engine will stop execution, makes the state of the execution to the database persistance and wait for the callback to occure.
As the long-running task implementation is not directly executed by Flowable's process engine and it does not participate in the process engine transaction, if there is an error in the service implementation, the failure will not cause the process engine to roll back.
The async job executor configuration is done in the src/main/resources/application.properties file. Simply, modify the following parameter to suit your needs:
The async job executor configuration is done by modifying two XML files (found under src/main/resources/):
task-queue-beans.xml: specified how each CWM's async job executor, dedicated to process a specific task is configuredtask-runnable-beans.xml: here you will find the runnable classes that are used by each CWM's async job executor to execute the task
As you can see, each queue used by the CWM's async job executor needs to have some configuration. Let's describe those parameters found in task-queue-beans.xml file.
runnableName: the id of the runnable that and instance is initialized and afterward executed. This is the id which relates to the information found intask-runnable-beans.xml.poolName: the name of the thread poolcorePoolSize: the number of threads to keep in the pool, even if they are idlemaximumPoolSize: the maximum number of threads to allow in the poolkeepAliveTimeInMillis: when the number of threads is greater than the core, this is the maximum time that excess idle threads will wait for new tasks before terminatingacquireWaitTimeInMillis: millis to wait before new tasks are pulled from the databasemaxTasksPerAcquisition: maximum tasks that can be pulled from the database
For instance task-queue-beans.xml:
<bean id="task1Queue" class="com.canchitodev.cwm.threadpool.service.TaskQueue" scope="prototype">
<property name="runnableName" value="task1Runnable" />
<property name="poolName" value="task1Queue" />
<property name="corePoolSize" value="2" />
<property name="maximumPoolSize" value="5" />
<property name="keepAliveTimeInMillis" value="300000"/>
<property name="acquireWaitTimeInMillis" value="5000"/>
<property name="maxTasksPerAcquisition" value="2"/>
</bean>For instance task-runnable-beans.xml:
<bean id="task1Runnable" class="com.canchitodev.cwm.tasks.runnable.Task1Runnable" scope="prototype"></bean>A long-running service task is composed of two different classes: _Behavio_r class and Runnable class. The behavior class, extends from AbstractTaskCanchitoBehavior, which at the same time extend from Flowable's TaskActivityBehavior class. Let's have a look at how to implement it by creating a new class:
@Service("task1")
@Scope("prototype")
public class Task1Behavior extends AbstractTaskCanchitoBehavior {
private static final long serialVersionUID = -4740654158860004620L;
@Override
public void execute(DelegateExecution execution) {
try {
this.submitTask(execution, new JSONObject(), "task1Runnable");
} catch (Exception e) {
this.throwException(execution,
"There was a problem when trying to execute task 'task1Runnable'"
);
}
}
@Override
protected void validateParameters(DelegateExecution execution) throws IllegalArgumentException {
// TODO Auto-generated method stub
}
}You can add additional logic before calling the submitTask() method if required. For our example, this would be enough. Next, create the Runnable class. Remember, this is the class with all the business logic.
public class Task1Runnable implements TaskRunnable {
private static final Logger logger = Logger.getLogger(Task1Runnable.class);
private GenericTaskEntity task;
public Task1Runnable() {}
public Task1Runnable(GenericTaskEntity task) {
this.task = task;
}
public GenericTaskEntity getTask() {
return task;
}
public void setTask(GenericTaskEntity task) {
this.task = task;
}
@Override
public void execute() {
try {
logger.info("Executing task " + task.toString());
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
logger.info("Done executing task " + task.toString());
}
}
}That's it. Now just add the task queue in the task-queue-beans.xml and the Runnable class to the respective task-runnable-beans.xml file so that TaskQueueService can start its respective AcquireTaskThread.