occasional pipeline failure caused by R4ExamplesDriver when inserting resource into derby embedded

[albertwang-ibm]

When the "all" index file is processed by R4ExamplesDriver in pipeline, from time to time, the pipeline fails with tons of exceptions like:

SEVERE: SQLException encountered while inserting Resource.
java.sql.SQLException: Too much contention on sequence FHIR_SEQUENCE. This is probably caused by an uncommitted scan of the SYS.SYSSEQUENCES catalog. Do not query this catalog directly. Instead, use the SYSCS_UTIL.SYSCS_PEEK_AT_SEQUENCE function to view the current value of a sequence generator.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedResultSet.closeOnTransactionError(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedResultSet.next(Unknown Source)
at com.ibm.fhir.persistence.jdbc.derby.DerbyResourceDAO.storeResource(DerbyResourceDAO.java:143)
at com.ibm.fhir.persistence.jdbc.dao.impl.ResourceDAOImpl.insertToDerby(ResourceDAOImpl.java:678)
at com.ibm.fhir.persistence.jdbc.dao.impl.ResourceDAOImpl.insert(ResourceDAOImpl.java:520)
at com.ibm.fhir.persistence.jdbc.impl.FHIRPersistenceJDBCImpl.create(FHIRPersistenceJDBCImpl.java:238)
at com.ibm.fhir.persistence.jdbc.test.spec.CreateOperation.process(CreateOperation.java:48)
at com.ibm.fhir.persistence.jdbc.test.spec.R4JDBCExamplesProcessor.process(R4JDBCExamplesProcessor.java:165)
at com.ibm.fhir.model.spec.test.R4ExamplesDriver.processExample(R4ExamplesDriver.java:463)
at com.ibm.fhir.model.spec.test.R4ExamplesDriver.processExample(R4ExamplesDriver.java:362)
at com.ibm.fhir.model.spec.test.R4ExamplesDriver.submitExample(R4ExamplesDriver.java:318)
...

Caused by: ERROR X0Y84: Too much contention on sequence FHIR_SEQUENCE. This is probably caused by an uncommitted scan of the SYS.SYSSEQUENCES catalog. Do not query this catalog directly. Instead, use the SYSCS_UTIL.SYSCS_PEEK_AT_SEQUENCE function to view the current value of a sequence generator.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.impl.sql.catalog.SequenceUpdater.tooMuchContentionException(Unknown Source)
at org.apache.derby.impl.sql.catalog.SequenceUpdater.getCurrentValueAndAdvance(Unknown Source)
at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getCurrentValueAndAdvance(Unknown Source)
at org.apache.derby.impl.sql.execute.BaseActivation.getCurrentValueAndAdvance(Unknown Source)
at org.apache.derby.exe.acbd0ea02fx016ex6b5ex6326x0000060270585.e0(Unknown Source)
at org.apache.derby.impl.services.reflect.DirectCall.invoke(Unknown Source)
at org.apache.derby.impl.sql.execute.RowResultSet.getNextRowCore(Unknown Source)
at org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(Unknown Source)
... 43 more

Nov 14, 2019 7:24:35 PM com.ibm.fhir.persistence.jdbc.impl.FHIRPersistenceJDBCImpl create
SEVERE: FK violation
com.ibm.fhir.persistence.jdbc.exception.FHIRPersistenceFKVException: SQLException encountered while inserting Resource. [probeId=a-1-0-4-64dc1c5b-c9d2-4aa2-aded-9902b5b4e610]
at com.ibm.fhir.persistence.jdbc.dao.impl.ResourceDAOImpl.insertToDerby(ResourceDAOImpl.java:705)
at com.ibm.fhir.persistence.jdbc.dao.impl.ResourceDAOImpl.insert(ResourceDAOImpl.java:520)
at com.ibm.fhir.persistence.jdbc.impl.FHIRPersistenceJDBCImpl.create(FHIRPersistenceJDBCImpl.java:238)
at com.ibm.fhir.persistence.jdbc.test.spec.CreateOperation.process(CreateOperation.java:48)
at com.ibm.fhir.persistence.jdbc.test.spec.R4JDBCExamplesProcessor.process(R4JDBCExamplesProcessor.java:165)
at com.ibm.fhir.model.spec.test.R4ExamplesDriver.processExample(R4ExamplesDriver.java:463)
at com.ibm.fhir.model.spec.test.R4ExamplesDriver.processExample(R4ExamplesDriver.java:362)
at com.ibm.fhir.model.spec.test.R4ExamplesDriver.submitExample(R4ExamplesDriver.java:318)