Remove flushCachesForGC from acquireExclusiveForGC path

[amicic]

flushCachesForGC is mostly called close/prior to start-gc point for each particual GC operation/STW-increment, so that we often end up doing it twice, the first one being early during acquireExclusiveVMAccessForGC negotiation.

This change removes the one from acquireExclusiveVMAccessForGC, but also adds a couple of flushCachesForGC in GC specific operation that relied on acquireExclusiveVMAccessForGC to do it.

[amicic]

@dmitripivkine still testing, but have a look

[amicic]

@dmitripivkine Ready for review, but can only be merged after eclipse-openj9/openj9#18950.

[dmitripivkine]

@babsingh Would you please to help to merge this item when eclipse-openj9/openj9#18950 is going to be promoted first?

[amicic]

Beside some code cleanup/simplification, it also slightly reduces the pause times, in order of 10s of micro seconds (dependent on the number of application threads).
Here is the effect on the first increment of Concurrent Scavenge (1.237ms vs 1.2ms).

grep -A20 '<cycle-start.scavenge' jbb_CS_PRESET9000_base.out |grep '<gc-op'|awk -F" '{x+=$6} END {print x/NR}'
1.23754
grep -A20 '<cycle-start.scavenge' jbb_CS_PRESET9000_noFlushOnExclusive.out |grep '<gc-op'|awk -F" '{x+=$6} END {print x/NR}'
1.19978

[babsingh]

jenkins build all

[babsingh]

@dmitripivkine eclipse-openj9/openj9#18950 has been merged. Are there any other dependencies or is this PR good to be merged?

[amicic]

there is no more dependancy

[babsingh]

OSX PR build failed due to a known and unrelated issue: #7181.