Inconsistent BatchStatus after database connection loss

[wrungel]

If during execution of a job the database connectivity gets lost and the job completes with error,
javax.batch.operations.JobOperator#getRunningExecutions still returns that job execution.

Environment:

• Wildfly 16.0.0.Final
• JDBC Job Repository
• A Business Application deployed in a WAR uses javax.batch.operations.JobOperator

Steps to reproduce:

1. Start Job X, note the job's execution ID
2. Before Job X completes, shutdown the database related to the JDBC Job Reporoistoty
3. Wait until Job X completes with error (as the Batch Runtime cannot commit Job's final state FAILED to the database)
4. Turn the database on
5. Call javax.batch.operations.JobOperator#getRunningExecutions(String jobName)
   Expected: javax.batch.operations.JobOperator#getRunningExecutions DOES NOT return execution ID from step 1
   Actual: javax.batch.operations.JobOperator#getRunningExecutions DOES return execution ID from step 1
6. Call javax.batch.operations.JobOperator#getJobExecution(long executionId)
   Expected: the returned JobExecution has the state FAILED
   Actual: the returned JobExecution has the state FAILED

[chengfang]

When running getRunningExecutions, the job operator tries to retrieve from the job repository database, which in this case contains stale status for certain job executions.

We could validate the status of the job executions retrieved from job repository against the cache. If the cached version contains more advanced status, then take the cached version as the correct status.