Drop Map Queries To Populate Boundary Test Loop's Local Cache

[AntsInMyEy3sJohnson]

Context

• Boundary test loop uses local cache to keep track of what it has written to target Hazelcast map in order to evaluate whether it is still within the given boundary
• Local cache re-initialized in each run prior to this run's operation chain
• Re-initializing local cache is performed via an SQL-like query against the remote Hazelcast cluster (to prevent the boundary test loop from querying keys that belong to other boundary test loops of the same Hazeltest instance, or boundary test loops of other Hazeltest instances, which could easily happen if the boundary test loop simply queried all keys)

Problem

• In Hazelcast cluster under high memory load, the map query may not return all values the test loop has written (for example, some of the previously written values might have already been evicted)
• The current implementation relies on that cache to be populated correctly -- if it is not, then the current operation chain is aborted, and re-initializing the cache is carried out again via the map query
• For large, un-indexed maps, these map queries can be very expensive for the target Hazelcast cluster to perform -- to the point where latencies on the map in question (set latency, in particular) skyrocket
• This could be seen as a feature -- Hazeltest is supposed to create load on the target Hazelcast cluster, after all --, but load should only be created as a result of explicit runner and test loop configuration, but never as a side effect of a runner executing a test loop

Solution

Find a way to work without synchronizing remote to local state. Perhaps managing state only locally without ever querying the Hazelcast cluster for it could do the trick with only minimal drawbacks.