Support AMD Ryzen?

[rocallahan]

Ryzen has a conditional branch counter. I have patches to use it here: https://github.com/mozilla/rr/tree/ryzen

To make it work reliably I had to increase the skid counter to 1000. That's pretty high, but OK. The patches make the skid size configurable per-architecture so we don't take that hit on Intel.

With these patches, most tests pass and the rest seem to be intermittent. In one run I get 10 failures out of 2068:

        327 - proc_fds (Failed)
        388 - ptracer_death_multithread_peer-no-syscallbuf (Failed)
        688 - condvar_stress-no-syscallbuf (Failed)
        835 - thread_stress (Failed)
        904 - checkpoint_prctl_name-no-syscallbuf (Failed)
        1362 - proc_fds-32-no-syscallbuf (Failed)
        1679 - async_signal_syscalls2-32 (Failed)
        1722 - condvar_stress-32-no-syscallbuf (Failed)
        1869 - thread_stress-32 (Failed)
        1870 - thread_stress-32-no-syscallbuf (Failed)

It appears that all these failures are due to intermittent overcounting. In most of them, during recording we seem to have overcounted a few conditional branches in the leadup to some syscall. In the rest, we seem to have overcounted during replay.

One interesting thing is that most of the syscalls where we detect the overcount are an mprotect (or a syscall following a syscall-buffered mprotect) that followed an mmap. There are two exceptions, one a read syscall and one a write syscall. I need to think about what this might mean.

[rocallahan]

I've written a testcase that just creates 100 do-nothing threads and then joins them all. Running 32 basic_test.run copies of that test in parallel usually means a few of them fail.

[rocallahan]

Some stats for 8 runs of 32 parallel tests each:

• 13 failures
• 6 made incorrect recordings, 7 failed during replay
• Of the 6 incorrect recordings, all of these recorded an overcount leading up to an mprotect

[rocallahan]

Here's a different run of 16 x 32 parallel tests:

• 33 failures
• 21 incorrect recordings
• 14 overcounts leading up to mprotect, 4 leading up to clone, 3 leading up to munmap

[rocallahan]

FWIW the executed syscall counts for one of those tests:

prctl 1
exit_group 1
arch_prctl 1
getrlimit 1
set_tid_address 1
rrcall_init_preload 1
execve 1
rt_sigprocmask 1
rt_sigaction 2
write 2
brk 3
geteuid 4
read 4
close 5
open 5
access 5
fstat 5
futex 82
munmap 97
exit 100
clone 100
madvise 100
set_robust_list 101
mprotect 110
mmap 114

[rocallahan]

And just for reference the syscalls before the overcount was detected are 14 mmap, 4 mprotect and 3 futex.

[rocallahan]

I tried writing a test that does a lot of mmap/mprotect/munmap in a loop and couldn't get it to fail much. When I put that loop in 10 parallel threads I'd still mostly get failures around thread creation.

[rocallahan]

Turns out has_kvm_in_txcp_bug was being set to true; setting IN_TXCP on AMD lets you start a counter, but it always returns 0, triggering count < NUM_BRANCHES. If I also require that count > 0 that avoids triggering the bug workarounds (namely the always_recreate_counters workaround) and then .... tada, 16 x 32 parallel tests, zero failures!!!

[rocallahan]

Now a lot of the tests are timing out. Which is weird. I'm also seeing some other issues I didn't see before.

[rocallahan]

Ah OK. When has_kvm_in_txcp_bug is false, we create a counter using IN_TXCP and use that to measure, and on AMD that just always returns 0. Which fixes overcounting just fine but has other issues :-).

[rocallahan]

I checked whether, if we create a counter with no interrupt set and one with an interrupt set, they always agree. They do, even when they overcount.

[rocallahan]

I tried creating three counters, one counting user-only events (U), one counting kernel events (K), and one counting both (A). You'd expect U + K = A, and that holds on Intel apparently, but on AMD it never does! A almost always has extra events, usually 8, sometimes 16, sometimes 1, once in a while a lot more...

[rocallahan]

FWIW those changes are not correlated with overcounts. So that's a dead end probably.

[rocallahan]

Also, that implies the problem is probably not an issue of kernel-mode events being incorrectly counted.

[rocallahan]

Interesting: there are a lot more failures running the 32-bit tests: 117 vs 39 (out of 8 x 32) just now.