Multi-conductor: in-flight workflow state lost when the primary stops mid-run

[vishalr-nrift]

I have been testing the XyOps conductors for the past few days and I need some help with the following problem.

Context:

• xyOps 1.0.87, two conductors - one primary and one online standby - over shared Postgres storage, with five xySat nodes listing both conductors.

• Conductor failover itself works well: the standby promotes in 10-25 seconds, the satellites re-attach, the schedule resumes, and nothing is executed twice.

• If a conductor stops mid-workflow, the job status of the event that was executing is lost, because it lives in heap memory and the record is only written to storage on completion.

• The consequence is that the step itself finishes normally on its satellite, but the next step never starts and the promoted primary shows no trace of the run - not running, not failed, not aborted.

  Questions:

1. Is this a genuine limitation of the multi-conductor design, or is there a setting or storage option we have missed?
2. Is there a supported workaround to carry an in-flight run across a failover, so the workflow continues on the surviving conductor?
3. Failing that, could the newly promoted primary at least mark workflows that were in progress as failed, with a reason? Our concern is the silence rather than the loss - an operator currently sees a workflow that appears to have stopped at step two when step three actually ran to completion.

[jhuckaby]

Oh wow, thank you for the report! This is actually an unintended regression bug. xyOps is supposed to automatically resume all active jobs and workflows on conductor failover.

It looks like the regression was introduced in xyOps v1.0.70, when I was working on improving network performance between conductor peers. This optimization code inadvertently wiped out the activeJobs object when a peer conductor loses a connection to its primary.

This has now been fixed and will be included in the next release.

I'm so sorry this bug was live for so long. It stayed hidden for 21 consecutive releases! How embarrassing. While I do have a full unit test suite, and run it before for every release, this particular case (multi-conductor failover with satellites and an active running workflow) is very difficult to test each and every time.

Anyway, thanks again! The next release will be out in just a few days.

[jhuckaby]

xyOps v1.0.92 was just released with a fix for this. Thank you again for reporting it!