Workflow persistence

Workflow execution state management

Persisting state of workflow execution

If you run a workflow with --persist flag, its execution state will be saved in a journal file, located in the workflow’s home directory, called <workflow_name>.<date>.log, where <workflow_name> is taken from the workflow.json file.

The log contains:

• Full workflow directory path
• hflow command line options used to run the workflow
• Input signals passed to the workflow
• Execution trace (firings of processes and output signals they produced)

Note that if a workflow function fails (never calls its callback), its entry will not be logged. You can use this behavior to recover workflow execution from a point it was interrupted, e.g. due to a failure.

Recovery

The following command runs a workflow recovering its state from the journal:

hflow recover <journal_file>

The workflow will continue running from the point where it was interrupted.

Such a recovered run is also persisted in a new journal file!

Invoking the function during recovery

By default, when a firing of a workflow activity is recovered, its workflow function is not invoked, simply the recovered outputs are reused. This behavior can be changed by setting flag executeWhenRecovering in the process’ metadata:

{
  "name": "Sum",
  "type": "dataflow",
  "function": "sum",
  "executeWhenRecovering": "true",   
  "ins": [ "square" ],
  "outs": [ "sum" ]
} 

If this flag is set, the function will always be invoked, even for recovered firings. Such an invocation differs from a normal one in that the function receives full instances of output signals produced in the previous run in the outs array. The function can tell if the invocation is "recovered" by examining the context.recovered flag.

The programmer can take advantage of this capability as follows:

function foo(ins, outs, context, cb) {
  if (context.recovered) {
      // recovery mode actions 
  } else {
     // normal execution
  }
}

For example, in the recovery mode the function may:

• examine the outs array and decide if it is necessary to re-execute the job or simply invoke cb(null, outs) without doing anything.
• re-compute internal state accumulated across multiple firings (stateful workflow activity)

Configuration of recovery

TODO