Basic Workflow: support for "stage"

[avernet]

Related issues

• Basic Workflow: perspective-based UI Basic Workflow: perspective-based UI #3463
• Basic Workflow: backend Relational persistence API to handle stage, action to set stage, function to read stage #4639

Use case

In a small business, anyone with the role employee can submit an expense report by filling out a form. After it has been submitted, the employee who created the expense report can can see but not edit the data (but other employees can't see the data), and anyone with the role manager can either approve or reject it.

New concept: stage

Overview

• What a stage is:
  • The set of possible stages a given form can be in is defined by the form author in Form Builder.
  • At any given point, a form is one of the stages specified by the form author. For instance, in the above use case, the form author would define the following stage: started, submitted, approved, rejected, with started marked as the initial stage.
• What the stage influences:
  • Different buttons can be shown at the bottom of the form depending on the current stage and the user's permissions (Ability to hide buttons based on permissions #1619). For instance, in the above use case, once submitted, the manager would see two buttons approve and reject.
  • Whether fields are read only or visible. For instance, in the above use case, when an expense report has been submitted, all the fields in the expense report would become read only.
• How the stage changes:
  • When a form is created, it gets the stage marked by the form author as the initial stage.
  • Buttons can change the current stage. For instance, in the above example the approve button will change the state to approved and the reject button will change the state to rejected.

In Form Builder

Stages

Out of the box, we'll have 2 stages: started (the default), and submitted. Our submit process will change the workflow stage to submitted, which will allow us to make a distinction in the database between form data that has been just saved, but hasn't been submitted yet, and data that has been submitted.

Available to

The Available to section shows in the Operations tab of the Workflow dialog. Its purpose is to allow form authors to whom a given operation is available, e.g. based on the user's authentication role or the workflow stage.

Pre-defined operations

Custom operations

Implementation

• Storage and persistence layer - On relational databases, the stage is stored in a separate column. On eXist, it is stored in the metadata.xml. For the REST API, the stage is sent off-band through headers, as done currently for say, roles and operations.
• Common code for permissions - Common code (proxy or Scala called by all persistence implementations) can determine, based on user's group and permissions, what the requirement on the stage is. Then, the persistence implementation can enforce that constraint, e.g. adding to its WHERE clause.
• New action to change stage - E.g. change-stage('approved').
• fr:stage() - A new XPath function is added that returns the current stage, so the stage can be used in MIPS and in processes where XPath expressions are used.
• Redirect to view - If the user goes to /edit and has only the right to view, then she is redirect to /view.

[Kevinhebertd]

Looking forward to this. Keep it up

[ebruchez]

@Kevinhebertd We are working on it :)

[ebruchez]

We have done quite a bit of progress on the UI. Remaining steps:

• UI (@ebruchez)
  • complete UI
  • hookup dialog UI to "Workflow Settings" button
  • open legacy permissions if exist, new workflow if exist
  • if legacy permissions, button to migrate to new permissions (once-only operation)
• build (@ebruchez)
  • fix sbt build
  • fix Travis build
• backend
  • decide how to serialize workflow config in form & implement (@ebruchez)
  • generate workflow config from existing permissions config
    • Workflow & Permissions in toolbox
  • generate permissions config from workflow config
    • NOTE: don't generate if user hasn't enabled workflow
    • Q: how exact can we be?
      • A: Couldn't we be exact, given that the database should test for the stage?
• persistence (@avernet)
  • support for Orbeon-Workflow-Stage header
  • store short permissions into metadata column / metadata.xml
  • when writing data, update stage in metadata.xml and stage column
  • check permissions for all operations (CRUD & search & published forms)
  • upgrade db schema & document
  • document API/db changes
• simple processes and functions
  • change-workflow-stage() action (store alongside mode & version)
  • fr:workflow-stage() function
• buttons
  • display buttons at bottom of page
  • handle legacy properties (document that they are used only if no workflow)
• redirect to view (see above)
• implement demo use cases
• document
• present new feature on orbeon.com

[Kevinhebertd]

Do you have an updated release date for 4.11 ?? 😃

[ebruchez]

It's not going to be in 2015, that's for sure. Shooting for sometime Q1 2016.

[ebruchez]

UPDATE: We have progressed on this until the end of 2015, but then decided that 2016.1 (AKA 4.11) would focus instead a number of features and fixes for customers. For 2016.2, we have decided to focus on one Form Builder usability feature.

Still, there was some work done:

• partial creation of the UI (see also Select or build Scala.js-based UI framework for Form Builder #2762) with Scala.js
• client/server communication with uPickle over our own protocol
• case class serialization/deserialization and store into form definition

When we pick this up, we should focus on addressing #2762 first.

[ebruchez]

Possible next steps:

• update UI to match new perspective-based design (Basic Workflow: perspective-based UI #3463)
• implement persistence changes
• function to migrate permissions to workflow

[avernet]

• As a first step towards moving where we check permissions (from the implementation of the persistence to Form Runner), we want to clarify the situations where in the future, when the new workflow is in place, we won't be able to always do that check at the level of the persistence API.

[avernet]

Consider the case of an expense report, and say that after an employee submits an expense report, the employee can update it, but not approve it, and a manager can approve it (or reject it), but not update it.

When a persistence API implementation gets a PUT, it doesn't have enough information to discriminate between an update of the initial expense report (only allowed if done by the employee) and an approval or rejection (only allowed if done by the manager). It doesn't have enough information to properly enforce the permissions.

It could reject operations that are clearly forbidden, like a POST by someone other than the employee or the manager. And this is something we could do in the future, but it has two downsides:

• It's more work (obviously, but should nevertheless be stated).
• It can provide a false sense of security, for instance leading one to think it is OK to expose the API to untrusted callers, say like a native application.

(It should be noted that this second point is already a problem right now: a form can implement the above scenario by hiding controls or making them readonly based on the user's role, and persistence API implementation doesn't check that.)

[ebruchez]

Summary of how do we handle the workflow UI so far:

• The workflow configuration is represented as a tree of case classes/objects
• We use Circe to encode/decode such trees to/from JSON.
• The XForms UI works on a simplified XML representation of the JSON-to-XML conversion.
• object WorkflowTemplates exposes to the UI templates for repeated content, etc. This is some amount of boilerplate.

The idea is that:

• The Scala case classes/objects are the source of truth for the user interface.
• The UI works on a structure which is isomorphic to that.

In practice, our XForms implementation requires XPath expressions pointing to XML. That is why we do this Scala ↔ XML translation. See also #3455 for similar considerations on how we could improve that.

[ebruchez]

I think we can continue along these lines for now. Steps:

• Update Scala definitions to match the requirements of the new UI.
• Update the UI code.
  • compare structure of old UI vs new
  • see what's reusable

[avernet]

Next steps:

• Write ADTs for stages (very simple), roles (very simple), buttons, and perspectives; let's call those "workflow ADTs" (same as first item in previous message by @ebruchez).
• Write serialization/deserialization of workflow ADTs
• read/write serialization to/from XML stored in the form definition.
• Write code converting current permissions → workflow ADTs (1-way conversion).
• Write code that figures what perspective applies, this based on the workflow ADTs, the current live document, the user's role.
• Write functions that, based on the current perspective tell if the user can perform given operation (access a given page), what buttons to show on the summary page, and what buttons to show on the detail page.
• Use functions mentioned in the previous point to check if users can access a page, and what buttons to show on the summary/detail page (for now we can keep using the oxf.fr.detail.button.BUTTON.enabled.*.* property to determine if the button is enabled or disabled).

[ebruchez]

Supporting native instances would be good, see #3455.

[ebruchez]

For reading/writing from/to the form definition:

Converting (option in the Form Builder permissions and/or Workflow dialogs?):

• if permissions are defined in the form
  • read permissions (PermissionsXML.parse())
  • convert them to WorkflowConfig (PermissionsToWorkflowConfig.convert())
  • serialize to XML
  • store under instance(fr-form-metadata)/workflow
  • include a format version (for example 2020.1)

Running:

• TODO

• Q: The "simplified XML" is useful right now for the UI. But should we store the definition to the form in JSON or XML? Right now we don't have a "simplified JSON".

• write Form Runner functions usable at runtime:

  • WorkflowConfig from XML

[ebruchez]

Following discussion with @avernet today, we suggest, for the stage name, to have a localizable label. This is usual for:

1. The Summary page
2. To display in the Detail page as well