2.7.8/3.0.1: BatchFetchType.IN causes IndexOutOfBoundsException in ForeignReferenceMapping.extractResultsFromBatchQuery

[jnehlmeier]

I have found https://bugs.eclipse.org/bugs/show_bug.cgi?id=412056 which describes the same issue, but even though it is marked as fixed for 2.6.x I can still reproduce the IndexOutOfBoundsException. The only workaround is to either set batch size very large so that a second batch query never has to be executed or using left fetch instead of batch fetch.

The main difference between the bug mentioned above and the code I used to reproduce the bug is a second level of batch fetches.

So instead of

class A {
  List<B> b_list;
}

class B {
  C c_attribute;
}

I use

class Assessment {
  Map<Long, Answer> answers;
}

class Answer {
  @ElementCollection
  Set<Tag> tags; // enum
}

with batch fetch hint assessment.answers.tags for a query SELECT assessment FROM Assessment assessment where assessment.id IN(...). The code then iterates in a randomized order through the result list of the query and first triggers batch fetch for assessment.answers and then batch fetch for answers.tags (which is the one that fails).

The concrete code can be seen in an example project that reproduces the issue:

Repo: https://github.com/CareCloud-GmbH/eclipselink-batch-fetch-bug
Main Class: https://github.com/CareCloud-GmbH/eclipselink-batch-fetch-bug/blob/main/src/main/java/eclipselink/bugs/batchfetch/App.java

When you use ./gradlew run it should launch the main class and you should see an IndexOutOfBoundsException. The code tries to produce the exception within 100 rounds of execution. It usually fails within the first few rounds but in the very rare case the code completes without exception, re-run the code. It sometimes succeeds because Collections.shuffle() does not guarantee to rearrange the iteration order in a way that causes the issue.

The output should look like

[main] INFO eclipselink.logging.all - EclipseLink, version: Eclipse Persistence Services - 3.0.1.v202104070723
.... insert queries ....
[main] INFO eclipselink.bugs.batchfetch.App - ======= ROUND 1 =======
... select queries ....
[main] INFO eclipselink.bugs.batchfetch.App - Processed Assessment[id=3]
... select queries ....
[main] INFO eclipselink.bugs.batchfetch.App - Processed Assessment[id=9]
... select queries ....
[main] INFO eclipselink.bugs.batchfetch.App - Processed Assessment[id=1]
... select queries ....
[main] INFO eclipselink.bugs.batchfetch.App - Processed Assessment[id=2]
... select queries ....
[main] ERROR eclipselink.bugs.batchfetch.App - Processing failed for Assessment[id=4]
Exception in thread "main" java.lang.IndexOutOfBoundsException: Index -1 out of bounds for length 12
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
	at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
	at java.base/java.util.Objects.checkIndex(Objects.java:359)
	at java.base/java.util.ArrayList.get(ArrayList.java:427)
	at org.eclipse.persistence.mappings.ForeignReferenceMapping.extractResultFromBatchQuery(ForeignReferenceMapping.java:592)
	at org.eclipse.persistence.mappings.CollectionMapping.extractResultFromBatchQuery(CollectionMapping.java:962)
	at org.eclipse.persistence.internal.indirection.BatchValueHolder.instantiate(BatchValueHolder.java:61)
	at org.eclipse.persistence.internal.indirection.QueryBasedValueHolder.instantiate(QueryBasedValueHolder.java:123)
	at org.eclipse.persistence.internal.indirection.DatabaseValueHolder.getValue(DatabaseValueHolder.java:97)
	at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiateImpl(UnitOfWorkValueHolder.java:175)
	at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiate(UnitOfWorkValueHolder.java:242)
	at org.eclipse.persistence.internal.indirection.DatabaseValueHolder.getValue(DatabaseValueHolder.java:97)
	at org.eclipse.persistence.indirection.IndirectSet.buildDelegate(IndirectSet.java:227)
	at org.eclipse.persistence.indirection.IndirectSet.getDelegate(IndirectSet.java:438)
	at org.eclipse.persistence.indirection.IndirectSet.size(IndirectSet.java:627)
	at eclipselink.bugs.batchfetch.App.loadDataAndTriggerBatchFetchIssue(App.java:96)
	at eclipselink.bugs.batchfetch.App.main(App.java:40)

The root cause is the same as in the issue mentioned above: The AbstractRecord (sourceRow) for the Answer that triggered the batch fetch using tags.size() is not available in BatchFetchPolicy.dataResults (originalPolicy). Thus the index search parentRows.indexOf(sourceRow) returns -1 and finally causes the exception in the first for loop (offset = -1, i = 0).

eclipselink/foundation/org.eclipse.persistence.core/src/main/java/org/eclipse/persistence/mappings/ForeignReferenceMapping.java

Lines 566 to 592 in 77a8c76

	if (originalPolicy.isIN()) {
	// Need to extract all foreign key values from all parent rows for IN parameter.
	List<AbstractRecord> parentRows = originalPolicy.getDataResults(this);
	// Execute queries by batch if too many rows.
	int rowsSize = parentRows.size();
	int size = Math.min(rowsSize, originalPolicy.getSize());
	if (size == 0) {
	return null;
	}
	int startIndex = 0;
	if (size != rowsSize) {
	// If only fetching a page, need to make sure the row we want is in the page.
	startIndex = parentRows.indexOf(sourceRow);
	}
	List foreignKeyValues = new ArrayList(size);
	Set foreignKeys = new HashSet(size);
	int index = 0;
	int offset = startIndex;
	for (int count = 0; count < size; count++) {
	if (index >= rowsSize) {
	// Processed all rows, done.
	break;
	} else if ((offset + index) >= rowsSize) {
	// If passed the end, go back to start.
	offset = index * -1;
	}
	AbstractRecord row = parentRows.get(offset + index);

[jnehlmeier]

Updated the linked project to use Eclipselink 3.1.0-M1 and the issue is still present. So I assume it is also still present in latest 2.7.x release.

Would be great if anybody could take a look at it.