[Jtreg/FFI]Fix the OOM issue with thunk allocation

[ChengJin01]

The changes aim to address the OOM issue with thunk allocation on the platforms with a small page size (e.g. 4KB) by maintaining a circular thunk heap list in which case a new thunk heap is created only when there is no free thunk heap available in the list to allocate thunk.

Signed-off-by: Cheng Jin jincheng@ca.ibm.com

[ChengJin01]

An intermitant OOM issue is detected in Jtreg FFI test suite at https://github.com/ibmruntimes/openj9-openjdk-jdk19/blob/openj9/test/jdk/java/foreign/TestUpcallAsync.java on zLinux and CentOS/x86_64 with pretty small page size (4KB), in which case the allocated thunks are held by native threads (created in native test code) for a while prior to termination. When the allocated thunks(to be released but delayed) accumulates, OOM is randomly triggered in upcall due to the temporary out-of-memory situation when allocating a new thunk for upcall.

This PR is to resolve this issue by replacing a single thunk heap with the fixed page size in the existing implementation with a circular linked list of thunk heaps so as to make the best of heaps for thunk allocation. The idea is straightforward as follows:

1. Initially there is only one heap(the head by default) in the list for thunk allocation, supposing it is called heap 1.
2. When heap 1 is out-of-memory, create a new heap 2 and insert it to the next one of heap 1 and set heap 2 as the list head.
3. When heap 2 is out-of-memory, move forward to heap 1 in the circular list, assuming some of allocated thunks are already released in heap 1. If heap 1 is free for allocation, set heap 1 as the list head to keep allocating thunks.
4. When there is no heap with free memory available in the list, create a new heap 3 and insert it to the next one of head to allocate thunks.

That being said, the list head always points to the free heap to be allocated till it is out-of-memory in which case we move to the next heap with free memory or create a new heap if there is no free memory for thunk in the list.

[ChengJin01]

Reviewer: @gacholio, @tajila
FYI: @pshipton, @knn-k

[ChengJin01]

Hi @knn-k, please help double-check to ensure pthread_jit_write_protect_np is still at the right place for Aarch64 in this PR.

[knn-k]

@ChengJin01 pthread_jit_write_protect_np for AArch64 macOS looks OK.

[gacholio]

Looks reasonable to me.

[gacholio]

This is complex enough that someone else should also look it over (suggesting @keithc-ca).

[keithc-ca]

@gacholio Can you please have another look at this? Thanks.

[gacholio]

What JDK do we need to test for these changes?

[ChengJin01]

What JDK do we need to test for these changes?

JDK17 & 19 (no difference in terms of native code in these changes).

[gacholio]

jenkins test sanity all jdk19

[gacholio]

jenkins compile win jdk8