Fix halted thread being allowed to continue execution v2

[amicic]

This PR is a variant of
#15201 with
handling of saveObjects NULL case

Primary issue: It was possible for a thread which is halted for
inspection to resume execution after a GC, resulting in the stack walk
crashing in the inspecting thread. This was introduced in
#12257 .

Secondary issue: If instrumentable object allocate is hooked, it was
possible to HCR at JIT object allocation points.

New restriction: The JVMTI extension event for instrumentable object
allocate may now only be acquired at startup, not during late attach. As
far as I know, there are no active users of this event.

Note that NOT_AT_SAFE_POINT has two distinct if related meanings:

• Normal exclusive VM access has priority over safe point

Setting NOT_AT_SAFE_POINT around all GCing operations accomplishes this.

• JIT optimization requires that there be no possibility of HCR at
  object allocation points

Extending the range of NOT_AT_SAFE_POINT to cover the entire allocation
path and disabling safe point if the instumentable object allocate event
is hooked ensures this.

This fix has several parts:

• Pause after GC if halt has been requested

After any possibly-GCing path, check to see if the thread should halt
before resuming mutation. This fixes the reported problem of inspected
threads continuing to run. This also requires that NOT_AT_SAFE_POINT be
set across the entirety of the allocation path to prevent safe point
from being acquired by the new halting code.

• Fix object allocation event reporting

If the instrumentable object allocation extension event was hooked,
there was a timing hole where HCR could occur at an object allocation
from the JIT, which is specificially what safe point HCR is meant to
avoid.

This is fixed by marking the thread NOT_AT_SAFE_POINT for the duration
of the allocate. If the instrumentable object allocate event is hooked,
disable safe point HCR as there is no way to safely report the event.

• Mark all possibly GCing paths NOT_AT_SAFE_POINT

This ensures that the GC has priority over safe point HCR.

haltThreadForInspection also now marks the inspecting thread NOT_SAFE
for the duration of the halt (see timing below).

Signed-off-by: Aleksandar Micic amicic@ca.ibm.com

[amicic]

Jenkins test sanity xLinux jdk11

[tajila]

Jenkins test sanity,extended plinux jdk8

[gacholio]

Thanks, was just signing in to do exactly this.