Calling cache_fifo::replace_which_way() directly results in a change in FIFO cache state

[bete0]

While working on a unit test for clients/drcachesim/simulator/cache_fifo.cpp (#5454), I discovered that calling cache_fifo::replace_which_way() right after accessing a cache line (by calling request()) would update the cache state.

For the cache replacement unit tests, we validate the expected way is replaced after an access by first sending a request() and then checking the way to be replaced on the next access matches the expected way as shown below (from #5454):

 void
 access_and_check_fifo(const addr_t addr,
                          const int expected_replacement_way_after_access)
 {
        memref_t ref;
        ref.data.type = TRACE_TYPE_READ;
        ref.data.size = 1;
        ref.data.addr = addr;
        request(ref);
        assert(replace_which_way(get_block_index(addr)) ==
                                 expected_replacement_way_after_access);
}

However calling replace_which_way() changes the state of the FIFO cache by updating the way to be replaced which results in the wrong way to be replaced on the next access.

This is not necessarily a bug in the FIFO cache replacement implementation but makes testing the implementation harder and potentially could result in unexpected behavior if cache_fifo::replace_which_way() is called directly somewhere else.

Steps to reproduce the behavior:

1. Checkout PR i#4842 drcachesim unit test: Add cache_fifo unit test #5454, build and run drcachesim unit test and observe the test fails
2. Update the test cases from https://github.com/DynamoRIO/dynamorio/pull/5454/files#diff-b7acbebc334f2800e156dcb14d7ad91ea8b243fee66b9cf3011430d2605081b9R165-R168 as follows and observe the test passes

  cache_fifo_test.access_and_check_fifo(ADDRESS_A, 1); // A x X X
  cache_fifo_test.access_and_check_fifo(ADDRESS_B, 3); // A B x X
  cache_fifo_test.access_and_check_fifo(ADDRESS_C, 1); // A B C x
  cache_fifo_test.access_and_check_fifo(ADDRESS_D, 3); // a B C D