BPMN / DMN Extensions and Support

Spiff Workflow - Release Notes for Version 1.0

After a year of extensive development to get SpiffWorkflow to support our effort at building a BPMN based workflow application, Sartography (our software development company) is pleased to announce an official release of Version 1.0 of SpiffWorkflow.

That's an auspicious jump from the previous 0.6, but we think it merits the version. We have used Spiff Workflow in roughly the present state for several months in a production environment supporting real users. We believe it is ready for this 1.0 moniker. We look forward to the rapid release of many subsequent versions over the coming years.

Here is a list of major enhancements:

1. Preserved Features - Most of the previously-existing tests were maintained, so these changes will hopefully still support any existing users. We’ve kept dependencies on external dependencies very low. And where we added things like Box and Levenshtein along the way, we pulled them back out, replacing them with internal implementations.

2. DMN Support - While it is not eye-watering perfect, we have a baseline of support for DMN processing that is fully integrated into the existing Execution Engine, so your DMN rules can be applied in the same language as your preferred execution language for BPMN expressions.

3. Forms - We’ve added support for Camunda-specific web forms, which are currently parsed and returned as JSON data structures. We’ve used this to generate Formly (an NPM package for Angular) forms for web pages and it works great, but we hope to extract it in the future and pass this through directly, to make SpiffWorkflow more general-purpose and allow just about any additional complex data to accompany a user task.

4. BPMN Additional Support - We made nearly innumerable enhancements to BPMN support. These, at a glance, include:

   • Full Serialization - A running BPMN/DMN-based workflow can now be fully serialized into JSON and restored to its original state from that JSON representation.
   • Multi-Instance Tasks - We can now handle multi-instance tasks of nearly any variety, including both Parallel and Sequential variants, as they occur in User Tasks, Script Tasks, or even Sub-Processes.
   • Pools & Lanes - We added support for BPMN pools and lanes which we used to successfully implement some complex approval processes in a production system.
   • Pluggable Script Engine - While our focus was on building a Python Script Engine, we added support for a broad set of FEEL expressions in the BPMN specification. We didn’t end up using them in production, as crossing languages proved to be an enormous headache for everyone, and the ability to write in Python for quick DMN expressions, and also complex scripts tasks proved too useful to deny. We assured that the same script engine is used throughout, and that it can be replaced by an alternate implementation by the calling application.
   • Visual Ordering of Tasks - When you have a series of parallel tasks that are aligned vertically in the diagram, that order is preserved in execution and in navigation data.
   • Sub-Workflows and Call Activities - We spent a lot of time improving the use of sub-workflows and call activities to assure it is easy to compose complex multi-file workflows that behave consistently and as expected.
   • Timer Events - Added ISO duration parsing for fine grained control over timed events.
   • Signals and Messages - Support for BPMN signals and listeners and adding the ability for the calling application to issue events - such as the “Cancel” event, so that workflows that are forced to stop can react before being terminated.
   • Boundary Events - For timed events, and messages, added support for BPMN events that alter the flow based on external events, with support for both interrupting and non-interrupting variants.
   • Looping Back - As BPMN diagrams can easily loop back to previous tasks, we put effort into testing this in SpiffWorkflow and resolving a number of performance and serialization issues related to the natural looping nature of BPMN.

5. Improved Logging and Better Errors - We spent a lot of time improving and standardizing error messages in Spiff so it is easier to tell what is going on where, when and why, particularly as you attempt to debug issues that occur in Script Tasks. We can even tell you the line in the script that failed, and offer “did you mean…” suggestions through a simple implementation of the Levenshtein distance algorithm. We also added the ability to halt a workflow at a specific engine_step, which can be useful if you want to run tests of your workflow, but have it stop at a specific moment.

6. Dot Notation - Our one major departure from Python syntax, we added the ability to use dot notation when referencing values deep within the Task Data, we found that using MyData.key was easier than MyData[“key”] for many areas in BPMN and DMN.
   Resets - We added the ability to reset the workflow to any previously completed task, and resume work from that point.

7. Navigation - We created a rich and complex navigation generator that is capable of describing any point in the tree of tasks. We found we desperately needed a way to tell where we were when parsing complex diagrams. While it didn’t prove terribly useful in the user interface so far, it’s a ludicrously-robust tool for debugging. At any moment, you can now print out the full data structure in a meaningful, visual way directly from the command line.

8. Tests - We added hundreds of tests, many of which load up example BPMN diagrams and clearly show expectations of how Spiff should behave in a large array of situations and edge cases.

9. Performance Improvements - As we used this in production we found numerous performance issues, which we have tried to address - mostly related to serializing and deserializing the workflows which happens a lot in a stateless web-based architecture. We added and iteratively refined some tools for gathering and reporting performance metrics, opening these up to the larger application, so you can debug and get a consistent log and tell if it is your app that is slow, or if things are slow within Spiff.

10. My God, It’s Full of Edge Cases - Some 40% of our commits are related to edge cases we encountered as we developed dozens of real-world BPMN/DMN-based workflows and fixing issues as we attempted to use complex BPMN elements together.

Dropping Support for Python 2:

We did need to remove a few items from Spiff along the way, the most notable is a drop in support for Python 2. Python 3.6, 3.7 and 3.8 are all supported and a part of our automated test suite.

Special Thanks:

Sartography (Sartography.com) undertook these efforts to support the development of a custom web application for the University of Virginia. We would like to thank UVA for their support and trust in allowing us to take on the mammoth task of building a general-purpose workflows system, and contributing something back to the open source community.

Bruce Silver, the author of BPMN Quick and Easy Using Method and Style, whose work we referenced extensively as we made implementation decisions and educated ourselves on the BPMN and DMN standards.

Samuel Abels (@knipknap) for keeping SpiffWorkflow alive for the past few years, and offering us commit access to make these contributions, and Matthew Hampton for kicking this project off and giving us sound footing on which to build.

Camunda, for whom, without their open source Modeler and the BPMN.js library, we would not have the tools to effectively build out our models, embed an editor in our application, and pull this mad mess together.

Kelly McDonald (@w4kpm) who dove deeper into the core of SpiffWorkflow than anyone else, and was instrumental in helping us get some of these major enhancements working correctly.

We would like to thank Denny Weinberg for his early contributions to DMN support, which we used as a baseline and then extended.

What is coming:

Clean-up: Moving forward, we want to drop direct support and parsing of Camunda-specific elements. We would like to pass non-BPMN parsing on to the calling application, allowing users to build plugin-based architectures that can address domain-specific objectives.

Cleaner Serialization: While our serialization and deserialization is rigorously tested, the JSON it produces is not easy to understand on its own. We would love to take time to clean this up, and make it easier to parse and extract data from a running workflow’s JSON representation.

Other Projects:

We are also working on a layered system architecture that will make the development of web-based applications for authoring and running BPMN-based workflows easier to accomplish. We’ve built a full REST based API on top of SpiffWorkflow for our pilot application, but it isn’t general enough to release. We’ve also created Javascript applications for both editing and running workflows specifications in a browser, but these still need to be generalized.

Our Dream:

A full stack of interconnected open source tools is the direction we are headed, along with a plug-in based architecture that will work seamlessly across that stack. We believe such grand ambitions are possible. Our ultimate ambition is the development of a Platform as a Service Business that facilitates the use of an increasingly standards compliant suite of powerful open source libraries and components.