Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support AMD Ryzen? #2034

Closed
rocallahan opened this issue May 27, 2017 · 273 comments · Fixed by #2671
Closed

Support AMD Ryzen? #2034

rocallahan opened this issue May 27, 2017 · 273 comments · Fixed by #2671

Comments

@rocallahan
Copy link
Collaborator

@rocallahan rocallahan commented May 27, 2017

Ryzen has a conditional branch counter. I have patches to use it here: https://github.com/mozilla/rr/tree/ryzen

To make it work reliably I had to increase the skid counter to 1000. That's pretty high, but OK. The patches make the skid size configurable per-architecture so we don't take that hit on Intel.

With these patches, most tests pass and the rest seem to be intermittent. In one run I get 10 failures out of 2068:

        327 - proc_fds (Failed)
        388 - ptracer_death_multithread_peer-no-syscallbuf (Failed)
        688 - condvar_stress-no-syscallbuf (Failed)
        835 - thread_stress (Failed)
        904 - checkpoint_prctl_name-no-syscallbuf (Failed)
        1362 - proc_fds-32-no-syscallbuf (Failed)
        1679 - async_signal_syscalls2-32 (Failed)
        1722 - condvar_stress-32-no-syscallbuf (Failed)
        1869 - thread_stress-32 (Failed)
        1870 - thread_stress-32-no-syscallbuf (Failed)

It appears that all these failures are due to intermittent overcounting. In most of them, during recording we seem to have overcounted a few conditional branches in the leadup to some syscall. In the rest, we seem to have overcounted during replay.

One interesting thing is that most of the syscalls where we detect the overcount are an mprotect (or a syscall following a syscall-buffered mprotect) that followed an mmap. There are two exceptions, one a read syscall and one a write syscall. I need to think about what this might mean.

@rocallahan
Copy link
Collaborator Author

@rocallahan rocallahan commented May 27, 2017

I've written a testcase that just creates 100 do-nothing threads and then joins them all. Running 32 basic_test.run copies of that test in parallel usually means a few of them fail.

@rocallahan
Copy link
Collaborator Author

@rocallahan rocallahan commented May 27, 2017

Some stats for 8 runs of 32 parallel tests each:

  • 13 failures
  • 6 made incorrect recordings, 7 failed during replay
  • Of the 6 incorrect recordings, all of these recorded an overcount leading up to an mprotect

@rocallahan
Copy link
Collaborator Author

@rocallahan rocallahan commented May 27, 2017

Here's a different run of 16 x 32 parallel tests:

  • 33 failures
  • 21 incorrect recordings
  • 14 overcounts leading up to mprotect, 4 leading up to clone, 3 leading up to munmap

@rocallahan
Copy link
Collaborator Author

@rocallahan rocallahan commented May 27, 2017

FWIW the executed syscall counts for one of those tests:

prctl 1
exit_group 1
arch_prctl 1
getrlimit 1
set_tid_address 1
rrcall_init_preload 1
execve 1
rt_sigprocmask 1
rt_sigaction 2
write 2
brk 3
geteuid 4
read 4
close 5
open 5
access 5
fstat 5
futex 82
munmap 97
exit 100
clone 100
madvise 100
set_robust_list 101
mprotect 110
mmap 114

@rocallahan
Copy link
Collaborator Author

@rocallahan rocallahan commented May 27, 2017

And just for reference the syscalls before the overcount was detected are 14 mmap, 4 mprotect and 3 futex.

@rocallahan
Copy link
Collaborator Author

@rocallahan rocallahan commented May 27, 2017

I tried writing a test that does a lot of mmap/mprotect/munmap in a loop and couldn't get it to fail much. When I put that loop in 10 parallel threads I'd still mostly get failures around thread creation.

@rocallahan
Copy link
Collaborator Author

@rocallahan rocallahan commented May 27, 2017

Turns out has_kvm_in_txcp_bug was being set to true; setting IN_TXCP on AMD lets you start a counter, but it always returns 0, triggering count < NUM_BRANCHES. If I also require that count > 0 that avoids triggering the bug workarounds (namely the always_recreate_counters workaround) and then .... tada, 16 x 32 parallel tests, zero failures!!!

@rocallahan
Copy link
Collaborator Author

@rocallahan rocallahan commented May 27, 2017

Now a lot of the tests are timing out. Which is weird. I'm also seeing some other issues I didn't see before.

@rocallahan
Copy link
Collaborator Author

@rocallahan rocallahan commented May 27, 2017

Ah OK. When has_kvm_in_txcp_bug is false, we create a counter using IN_TXCP and use that to measure, and on AMD that just always returns 0. Which fixes overcounting just fine but has other issues :-).

@rocallahan
Copy link
Collaborator Author

@rocallahan rocallahan commented May 27, 2017

I checked whether, if we create a counter with no interrupt set and one with an interrupt set, they always agree. They do, even when they overcount.

@rocallahan
Copy link
Collaborator Author

@rocallahan rocallahan commented May 27, 2017

I tried creating three counters, one counting user-only events (U), one counting kernel events (K), and one counting both (A). You'd expect U + K = A, and that holds on Intel apparently, but on AMD it never does! A almost always has extra events, usually 8, sometimes 16, sometimes 1, once in a while a lot more...

@rocallahan
Copy link
Collaborator Author

@rocallahan rocallahan commented May 27, 2017

FWIW those changes are not correlated with overcounts. So that's a dead end probably.

@rocallahan
Copy link
Collaborator Author

@rocallahan rocallahan commented May 27, 2017

Also, that implies the problem is probably not an issue of kernel-mode events being incorrectly counted.

@rocallahan
Copy link
Collaborator Author

@rocallahan rocallahan commented May 27, 2017

Interesting: there are a lot more failures running the 32-bit tests: 117 vs 39 (out of 8 x 32) just now.

@rocallahan
Copy link
Collaborator Author

@rocallahan rocallahan commented May 27, 2017

By spraying rdtscs around syscall entry points to create trace events, I can see that the overcounts are detected at the rdtsc before mprotect typically. That suggests they're occurring during user-mode execution or during any entry to the kernel, not specifically related to system calls. Maybe it's a hardware issue that happens to be easier to trigger when doing thread creation.

I'm out of ideas. It appears that the Ryzen PMU just isn't quite accurate enough :-(. rr might work OK for some kinds of usage but I wouldn't recommend it.

I'll land the patches I have with a warning for Ryzen users that things won't be reliable.

@gabrielesvelto
Copy link

@gabrielesvelto gabrielesvelto commented Jun 26, 2018

FYI AMD has posted an errata for its Ryzen CPUs and it includes multiple issues with performance counters, namely:

  • 990: Certain Performance Counters For Retire Based Events May Overcount
  • 1023: Performance Monitor Counter Overflow Interrupts May Fail To Be Delivered When Two or More Counters Are Enabled
  • 1047: Miss Address Buffer Performance Counter May Be Inaccurate
  • 1054: Instructions Retired Performance Counter May Be Inaccurate
  • 1067: L3 Performance Event Counter May Be Inaccurate

None of them involves PMCx0D1 directly (which I believe is what rr uses). Either way none of them has a planned fix or suggested workaround.

@rocallahan
Copy link
Collaborator Author

@rocallahan rocallahan commented Aug 24, 2018

Of those errata:

  • as Gabriele noted, 990, 1047, 1054 and 1067 do not include retired conditional branches (0xD1)
  • 1023 does not affect rr since we only program one counter for interrupts

@rocallahan
Copy link
Collaborator Author

@rocallahan rocallahan commented Sep 20, 2018

The patches in PR #2255 might work on Ryzen. It would be great if someone could test. You'll have to change PerfCounters.cc to to assign the same configuration to AMDRyzen as AMDF15R30.

@rocallahan
Copy link
Collaborator Author

@rocallahan rocallahan commented Sep 20, 2018

Here is the patch to test with: cdf4e27
Building and testing instructions here: https://github.com/mozilla/rr/wiki/Building-And-Installing

@pipcet
Copy link
Contributor

@pipcet pipcet commented Sep 20, 2018

I just checked the Bios and Kernel Developers' Guides on the AMD page, and all recent AMD CPUs appear to have PMCs 0xc4 and 0xc6; of course that doesn't mean they're reliable, but it might be worth checking this on all those we can still find users of.

@rocallahan
Copy link
Collaborator Author

@rocallahan rocallahan commented Sep 21, 2018

I get 57 test failures on Ryzen (AMD EPYC 7401P 24-Core Processor), but the alarm tests seem to work so there's hope.

@rocallahan
Copy link
Collaborator Author

@rocallahan rocallahan commented Sep 21, 2018

Looks like Ryzen still doesn't work. I landed a join_threads test on master that starts 100 threads and then joins them. Running

(for i in `seq 1 200`; do bash basic_test.run join_threads & done; wait)>& /tmp/output

I get 10+ ticks mismatch errors every run.

@pipcet it would be great if you can run that test yourself to make sure you don't get any errors on your Bulldozer machine.

@rocallahan
Copy link
Collaborator Author

@rocallahan rocallahan commented Sep 21, 2018

That is run from the rr test directory of course.

@rocallahan
Copy link
Collaborator Author

@rocallahan rocallahan commented Sep 21, 2018

The symptoms are similar to the issues I saw with the conditional branches approach before, so it's possible that at some point after Bulldozer AMD introduced a bug that destablized multiple types of counters.

@pipcet
Copy link
Contributor

@pipcet pipcet commented Sep 21, 2018

The join_threads test you posted passes all 200 runs here.

Some ideas:

  • are you using simultaneous multi-threading? I don't know whether the PMCs are per-thread or per-core on Ryzen (the documentation seems to suggest per-thread, though)
  • regardless of SMT, it might be worth it to try downcoring to just one CPU
  • we can try PMC C2 instead of C4; there's also an outside chance that C0 or C1 work.

@pipcet
Copy link
Contributor

@pipcet pipcet commented Sep 21, 2018

There's also section 2.1.11.2 in https://developer.amd.com/wp-content/resources/56255_3_03.PDFm/wp-content/resources/56255_3_03.PDF, which I don't think applies here:

An option is provided for merging a pair of even/odd performance monitors to acquire an accurate count.

However, that would probably require kernel hacking...

@eddyb
Copy link

@eddyb eddyb commented Sep 20, 2020

@tuxiqae Thanks for confirming, it looks like lock speculation is indeed disabled and you should be able to use rr without any workarounds.

@tuxiqae
Copy link

@tuxiqae tuxiqae commented Sep 20, 2020

@tuxiqae Thanks for confirming, it looks like lock speculation is indeed disabled and you should be able to use rr without any workarounds.

Amazing! As soon as I finish with this kernel I will upload the logs for the latest one and let you guys know what it looks like.

Thanks for the help and for doing so much for the community :D

@eddyb
Copy link

@eddyb eddyb commented Sep 20, 2020

One thing I wanted to bring up is that AMD did some weird stuff with the documentation for the PMCx025 (Retired Lock Instructions) counters (which I thought might explain why @tuxiqae was seeing those 0 results, but that was a red herring).

Three sources for these "PPR"(Processor Programming Reference) documents were used:


Here are the categories (to be clear, we have no evidence to suggest there is any difference between these, everything we've tested on behaves consistently, they're just documented weirdly):

Older names (BusLock, NonSpecLock, SpecLock, SpecLockMapCommit)

image

All bits reserved

image

Only BusLock (and 0x0e Unit Mask for everything else) documented

image

Newer names (BusLock, NonSpecLock, SpecLockLoSpec, SpecLockHiSpec)

image

I'm not sure what Lo/Hi are, as the magnitudes observed wouldn't make sense if they were halves of a larger register, my best guess is they're different kinds of speculation, which would match up with SpecLock vs SpecLockMapCommit.

Note that SpecLock aka SpecLockLoSpec keeps counting even when SpecLockMap is turned off (via MSRC001_1020[54]), but stops counting when that MSRC001_1020[8] bit is set, which is why I suspect that's the older/simpler/more general "lock speculation" mode being turned off.

@tuxiqae
Copy link

@tuxiqae tuxiqae commented Sep 20, 2020

@eddyb

15:35:37 ❯ uname -r
5.8.10-arch1-1
15:36:18 ❯ head -n 6 /proc/cpuinfo
processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 23
model		: 1
model name	: AMD Ryzen 7 1800X Eight-Core Processor
stepping	: 1
15:37:00 ✖ rustup component add --toolchain nightly rustc-dev && curl -sS https://gist.githubusercontent.com/eddyb/269491b0b59605f39f1d0a9dcf535c4a/raw/73d6a9c55783d52bfbc8cc1ae6b7e8300a9ef7c9/rdpmc-bench.rs | rustc +nightly - -o rdpmc-bench -O && ./rdpmc-bench
info: component 'rustc-dev' for target 'x86_64-unknown-linux-gnu' is up to date
warning: field is never read: `start_instructions_minus_interrupts`
  --> <anon>:85:9
   |
85 |         start_instructions_minus_interrupts: u64,
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: `#[warn(dead_code)]` on by default

warning: associated function is never used: `instructions_since_start`
   --> <anon>:112:23
    |
112 |         pub(super) fn instructions_since_start(&self) -> u64 {
    |                       ^^^^^^^^^^^^^^^^^^^^^^^^

warning: 2 warnings emitted

Counter::new: version=0 compat_version=0 index=0x2
CpuModel::detect: vendor="AuthenticAMD" family=23 model=1
CpuModel::detect: known AMD CPU: Zen (Naples/Whitehaven/Summit Ridge/Snowy Owl)
Counter::new: version=0 compat_version=0 index=0x3
Counter::new: version=0 compat_version=0 index=0x3

# `rdpmc` benchmarks: `lock xadd` noise

### `100000` iterations
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)

total=0

### `1000000` iterations
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)

total=0

rdpmc-bench instructions:u=550047535 hw_interrupts.received:u=86

Again, I run ./rdpmc-bench a couple of times, of which that's the latest output.

15:37:30 ❯ ./rdpmc-bench
Counter::new: version=0 compat_version=0 index=0x2
CpuModel::detect: vendor="AuthenticAMD" family=23 model=1
CpuModel::detect: known AMD CPU: Zen (Naples/Whitehaven/Summit Ridge/Snowy Owl)
Counter::new: version=0 compat_version=0 index=0x3
Counter::new: version=0 compat_version=0 index=0x3

# `rdpmc` benchmarks: `lock xadd` noise

### `100000` iterations
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)

total=0

### `1000000` iterations
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)
0 (50 * 0)

total=0

rdpmc-bench instructions:u=550047532 hw_interrupts.received:u=83

ctest-latest-kernel.txt

@eddyb
Copy link

@eddyb eddyb commented Sep 20, 2020

Looks like your kernel wasn't responsible for any of the effects after all, my bad.

Btw you can edit your comments to use this:

<details>
<summary>rdpmc-bench output (click to open)</summary>

...
</details>

(where the ... is where you place the long output. the empty line after <summary>...</summary> is so that it can switch back into markdown mode after the HTML)

@bnjbvr
Copy link

@bnjbvr bnjbvr commented Sep 20, 2020

Hi! Thanks for working on this, this is really awesome!

I can confirm that before running the zen_workaround script, ctests indicated mostly failures. After successfully running the script (it says the workaround is in place), I get only one test failure in 4 runs (details below: there are two intermittent failures). The rdpmc program runs correctly and displayed only zeroes in every of the 10 runs I did.

Most importantly, I could record and replay a run of a Spidermonkey JS test failure, that included reverse-stepping around voluntarily-buggy JIT assembly code, so I can now do my job with this new RR \o/ Thanks a bunch to y'all!

uname + cpuinfo
ben@bertka-linux [~] 
➜ uname -r
5.4.0-47-generic
ben@bertka-linux [~] 
➜ head -n 6 /proc/cpuinfo
processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 23
model		: 113
model name	: AMD Ryzen 9 3950X 16-Core Processor
stepping	: 0
`ctest -j$(nproc)`

Seen in 3/4 runs:

 557/2487 Test  #565: setuid-no-syscallbuf ............................................***Failed  Error regular expression found in output. Regex=[FAILED]  0.36 sec

[...]

99% tests passed, 1 tests failed out of 2487

Total Test time (real) = 206.20 sec

The following tests FAILED:
	565 - setuid-no-syscallbuf (Failed)
Errors while running CTest

Seen in 1/4 runs:

99% tests passed, 1 tests failed out of 2487

Total Test time (real) = 155.20 sec

The following tests FAILED:
	2339 - checkpoint_mmap_shared-32-no-syscallbuf (Failed)

@emilio
Copy link
Contributor

@emilio emilio commented Sep 20, 2020

@bnjbvr the output of ctest --verbose -j$(nproc) would be useful to debug those failures I suspect.

@bnjbvr
Copy link

@bnjbvr bnjbvr commented Sep 20, 2020

Good idea!

Failure of 565
565: Test command: /usr/bin/bash "source_dir/src/test/basic_test.run" "setuid" "-n" "bin_dir" "120"
565: Test timeout computed to be: 1000
565: source_dir/src/test/util.sh: line 258: 1003481 Aborted                 (core dumped) _RR_TRACE_DIR="$workdir" test-monitor $TIMEOUT replay.err $RR_EXE $GLOBAL_OPTIONS replay -a $replayflags > replay.out 2> replay.err
565: Test 'setuid' FAILED: : error during replay:
565: --------------------------------------------------
565: [ERROR /home/ben/code/rr/src/Registers.cc:295:maybe_print_reg_mismatch()] r10 0x55ca0684995a != 0x55ca0684995b (replaying vs. recorded)
565: process 1003489 sent SIGURG
565: ====== /proc/1003489/status
565: Name:	rr
565: Umask:	0002
565: State:	S (sleeping)
565: Tgid:	1003489
565: Ngid:	0
565: Pid:	1003489
565: PPid:	1003481
565: TracerPid:	0
565: Uid:	1000	1000	1000	1000
565: Gid:	1000	1000	1000	1000
565: FDSize:	64
565: Groups:	27 136 1000 
565: NStgid:	1003489
565: NSpid:	1003489
565: NSpgid:	1003489
565: NSsid:	624841
565: VmPeak:	   11772 kB
565: VmSize:	   11260 kB
565: VmLck:	       0 kB
565: VmPin:	       0 kB
565: VmHWM:	    7888 kB
565: VmRSS:	    7388 kB
565: RssAnon:	    1052 kB
565: RssFile:	    6336 kB
565: RssShmem:	       0 kB
565: VmData:	    1104 kB
565: VmStk:	     132 kB
565: VmExe:	    2340 kB
565: VmLib:	    3832 kB
565: VmPTE:	      64 kB
565: VmSwap:	       0 kB
565: HugetlbPages:	       0 kB
565: CoreDumping:	0
565: THP_enabled:	1
565: Threads:	1
565: SigQ:	1/127941
565: SigPnd:	0000000000000000
565: ShdPnd:	0000000000000000
565: SigBlk:	0000000000000000
565: SigIgn:	0000000000000000
565: SigCgt:	0000000180002000
565: CapInh:	0000000000000000
565: CapPrm:	0000000000000000
565: CapEff:	0000000000000000
565: CapBnd:	0000003fffffffff
565: CapAmb:	0000000000000000
565: NoNewPrivs:	0
565: Seccomp:	0
565: Speculation_Store_Bypass:	thread vulnerable
565: Cpus_allowed:	00800000
565: Cpus_allowed_list:	23
565: Mems_allowed:	00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
565: Mems_allowed_list:	0
565: voluntary_ctxt_switches:	211
565: nonvoluntary_ctxt_switches:	129
565: ====== /proc/1003489/stack
565: ====== /proc/1003537/status
565: Name:	rr:setuid-B75X8
565: Umask:	0002
565: State:	t (tracing stop)
565: Tgid:	1003537
565: Ngid:	0
565: Pid:	1003537
565: PPid:	1003489
565: TracerPid:	1003489
565: Uid:	1000	1000	1000	1000
565: Gid:	1000	1000	1000	1000
565: FDSize:	1024
565: Groups:	27 136 1000 
565: NStgid:	1003537
565: NSpid:	1003537
565: NSpgid:	1003537
565: NSsid:	1003537
565: VmPeak:	    5200 kB
565: VmSize:	    5096 kB
565: VmLck:	       0 kB
565: VmPin:	       0 kB
565: VmHWM:	    2096 kB
565: VmRSS:	    2096 kB
565: RssAnon:	     384 kB
565: RssFile:	    1712 kB
565: RssShmem:	       0 kB
565: VmData:	    2460 kB
565: VmStk:	       0 kB
565: VmExe:	       8 kB
565: VmLib:	    1924 kB
565: VmPTE:	      56 kB
565: VmSwap:	       0 kB
565: HugetlbPages:	       0 kB
565: CoreDumping:	0
565: THP_enabled:	1
565: Threads:	1
565: SigQ:	1/127941
565: SigPnd:	0000000000000000
565: ShdPnd:	0000000000000000
565: SigBlk:	0000000000000000
565: SigIgn:	0000000000010000
565: SigCgt:	0000000000000000
565: CapInh:	0000000000000000
565: CapPrm:	0000000000000000
565: CapEff:	0000000000000000
565: CapBnd:	0000003fffffffff
565: CapAmb:	0000000000000000
565: NoNewPrivs:	1
565: Seccomp:	0
565: Speculation_Store_Bypass:	thread vulnerable
565: Cpus_allowed:	00800000
565: Cpus_allowed_list:	23
565: Mems_allowed:	00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
565: Mems_allowed_list:	0
565: voluntary_ctxt_switches:	332
565: nonvoluntary_ctxt_switches:	1
565: ====== /proc/1003537/stack
565: ====== gdb -p 1003489 -ex 'set confirm off' -ex 'set height 0' -ex 'thread apply all bt' -ex q </dev/null 2>&1
565: GNU gdb (Ubuntu 9.1-0ubuntu1) 9.1
565: Copyright (C) 2020 Free Software Foundation, Inc.
565: License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
565: This is free software: you are free to change and redistribute it.
565: There is NO WARRANTY, to the extent permitted by law.
565: Type "show copying" and "show warranty" for details.
565: This GDB was configured as "x86_64-linux-gnu".
565: Type "show configuration" for configuration details.
565: For bug reporting instructions, please see:
565: <http://www.gnu.org/software/gdb/bugs/>.
565: Find the GDB manual and other documentation resources online at:
565:     <http://www.gnu.org/software/gdb/documentation/>.
565: 
565: For help, type "help".
565: Type "apropos word" to search for commands related to "word".
565: Attaching to process 1003489
565: Could not attach to process.  If your uid matches the uid of the target
565: process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try
565: again as the root user.  For more details, see /etc/sysctl.d/10-ptrace.conf
565: ptrace: Operation not permitted.
565: --------------------------------------------------
565: replay.out:
565: --------------------------------------------------
565: --------------------------------------------------
565: Test setuid failed, leaving behind /tmp/rr-test-setuid-B75X8dNJX
565: To replay the failed test, run
565:   _RR_TRACE_DIR=/tmp/rr-test-setuid-B75X8dNJX rr replay

(I'll edit this post if i manage to reproduce the second one; might have been a timeout)

@bjorn3
Copy link

@bjorn3 bjorn3 commented Sep 20, 2020

565: Attaching to process 1003489
565: Could not attach to process. If your uid matches the uid of the target
565: process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try
565: again as the root user. For more details, see /etc/sysctl.d/10-ptrace.conf
565: ptrace: Operation not permitted.

Could this be the problem? What is the value of /proc/sys/kernel/yama/ptrace_scope? Does it work if you write 0 to it? A value of 1 has the following effect according to man ptrace: "When performing an operation that requires a PTRACE_MODE_ATTACH check, the calling process must either have the CAP_SYS_PTRACE capability in the user namespace of the target process or it must have a predefined relationship with the target process. By default, the predefined relationship is that the target process must be a descendant of the caller.".

@bnjbvr
Copy link

@bnjbvr bnjbvr commented Sep 20, 2020

Could this be the problem?

It's not conclusive. I've had the test intermittently pass with /proc/sys/kernel/yama/ptrace_scope set to 1, and the test both intermittently pass or fail with /proc/sys/kernel/yama/ptrace_scope set to 0.

@khuey
Copy link
Collaborator

@khuey khuey commented Sep 20, 2020

565: [ERROR /home/ben/code/rr/src/Registers.cc:295:maybe_print_reg_mismatch()] r10 0x55ca0684995a != 0x55ca0684995b (replaying vs. recorded) is the interesting line. The yama ptrace thing is just preventing us from attaching gdb to get a stack trace.

There should be a trace in /tmp/rr-test-setuid-B75X8dNJX after this. Does replaying that same trace always fail with this error?

@tuxiqae
Copy link

@tuxiqae tuxiqae commented Sep 20, 2020

Looks like your kernel wasn't responsible for any of the effects after all, my bad.

Btw you can edit your comments to use this:

<details>
<summary>rdpmc-bench output (click to open)</summary>

...
</details>

(where the ... is where you place the long output. the empty line after <summary>...</summary> is so that it can switch back into markdown mode after the HTML)

Is there anything else you'd like me to test?

@rocallahan
Copy link
Collaborator Author

@rocallahan rocallahan commented Sep 20, 2020

Good idea!
Failure of 565

(I'll edit this post if i manage to reproduce the second one; might have been a timeout)

Please spin off this 565 failure into its own issue.

EDIT: I did it myself: #2694

@eddyb
Copy link

@eddyb eddyb commented Sep 21, 2020

@tuxiqae Not from me, but you should get some guidance from @glandium or @rocallahan about your failures. Btw you didn't need to post the whole logs, just the list of failed tests, at the bottom, e.g. this was the only relevant part from your "latest kernel":

99% tests passed, 14 tests failed out of 2487

Total Test time (real) = 1000.28 sec

The following tests FAILED:
	698 - x86/sysfs (Failed)
	699 - x86/sysfs-no-syscallbuf (Failed)
	1153 - record_replay-no-syscallbuf (Failed)
	1496 - legacy_ugid-32 (Failed)
	1497 - legacy_ugid-32-no-syscallbuf (Failed)
	1814 - shm-32 (Failed)
	1815 - shm-32-no-syscallbuf (Failed)
	1942 - x86/sysfs-32 (Failed)
	1943 - x86/sysfs-32-no-syscallbuf (Failed)
	1976 - unshare-32 (Timeout)
	1977 - unshare-32-no-syscallbuf (Failed)
	2394 - record_replay-32 (Failed)
	2436 - shm_checkpoint-32 (Failed)
	2437 - shm_checkpoint-32-no-syscallbuf (Failed)

FWIW, @mati865 (on Ryzen 1600) and @nagisa (on Ryzen 1700) ran my counter test, and those CPUs don't behave any differently from any other Zen (1, 1+ or 2) ones, so I guess the MSRC001_1020[8] bit isn't commonly set. They haven't ran rr's tests on those machines yet though.

Okay, @tuxiqae, there is one more thing: if you don't mind me asking, what's your motherboard? Because that's the only source of MSRC001_1020[8] being set that I can think of, if it's the same for both kernels.

@khuey
Copy link
Collaborator

@khuey khuey commented Sep 21, 2020

@tuxiqae Not from me, but you should get some guidance from @glandium or @rocallahan about your failures. Btw you didn't need to post the whole logs, just the list of failed tests, at the bottom, e.g. this was the only relevant part from your "latest kernel":

99% tests passed, 14 tests failed out of 2487

Total Test time (real) = 1000.28 sec

The following tests FAILED:
	698 - x86/sysfs (Failed)
	699 - x86/sysfs-no-syscallbuf (Failed)
	1153 - record_replay-no-syscallbuf (Failed)
	1496 - legacy_ugid-32 (Failed)
	1497 - legacy_ugid-32-no-syscallbuf (Failed)
	1814 - shm-32 (Failed)
	1815 - shm-32-no-syscallbuf (Failed)
	1942 - x86/sysfs-32 (Failed)
	1943 - x86/sysfs-32-no-syscallbuf (Failed)
	1976 - unshare-32 (Timeout)
	1977 - unshare-32-no-syscallbuf (Failed)
	2394 - record_replay-32 (Failed)
	2436 - shm_checkpoint-32 (Failed)
	2437 - shm_checkpoint-32-no-syscallbuf (Failed)

This is essentially the same failure set as #2681.

@tuxiqae
Copy link

@tuxiqae tuxiqae commented Sep 21, 2020

Okay, @tuxiqae, there is one more thing: if you don't mind me asking, what's your motherboard? Because that's the only source of MSRC001_1020[8] being set that I can think of, if it's the same for both kernels.

My motherboard is Gigabyte AX370-Gaming K7 version F31

@eddyb
Copy link

@eddyb eddyb commented Sep 21, 2020

(when quoting it would be a good idea to not include all the log snippets because they add up. though it might not matter, I think I'll copy my comments with information dumps into a larger report or something, eventually, to avoid linking to this thread)

@eddyb
Copy link

@eddyb eddyb commented Sep 21, 2020

@khuey thanks! I Ctrl+F'd but ofc I didn't have most of the comments loaded because there's been so many.

@tuxiqae If you missed it above (#2034 (comment)), @khuey said all your test failures are the ones in #2681, and if I had to guess, they're probably related to your more recent kernel version (doesn't mean you should use an older kernel, I think the plan is for rr to work around these issues).

@tuxiqae
Copy link

@tuxiqae tuxiqae commented Sep 21, 2020

So as of now it is bound to fail I guess, I'll just wait for an update.
Thanks!

@eddyb
Copy link

@eddyb eddyb commented Sep 21, 2020

You may still be able to use rr, I think, as long as the application being recorded doesn't happen to use those features.

@khuey
Copy link
Collaborator

@khuey khuey commented Sep 21, 2020

Those failures are unlikely to impact a real application. They just affect our tests (which try to exercise the entire syscall space, including syscalls that may not exist or work properly on your kernel/libc).

@rocallahan
Copy link
Collaborator Author

@rocallahan rocallahan commented Sep 21, 2020

This is essentially the same failure set as #2681.

The test issues in #2681 are fixed on master for a couple of days now.

@glandium
Copy link
Contributor

@glandium glandium commented Sep 21, 2020

Just to add to @eddyb's information, MSRC001_1020[8] does work as a workaround instead of MSRC001_1020[54], and the current rr detection code as to whether a workaround needs to be applied or not has the right behavior (not showing a warning when either MSR bit is set). Considering the wiki page tells to apply the workaround only when rr shows the warning, I don't think we need to adjust the workaround script itself.

@tuxiqae
Copy link

@tuxiqae tuxiqae commented Sep 21, 2020

@rocallahan

This is essentially the same failure set as #2681.

The test issues in #2681 are fixed on master for a couple of days now.

I'm currently updated to commit 8fc7d2a but still see those errors.. 🤔

@rocallahan
Copy link
Collaborator Author

@rocallahan rocallahan commented Sep 21, 2020

Please file a new issue and give us the verbose logs for those test failures. Thanks!

@vchuravy
Copy link

@vchuravy vchuravy commented Sep 28, 2020

I see that there is a kernel module available, is there a kernel patch flying around to try out?

bkin pushed a commit to bkin/rr that referenced this issue May 20, 2021
szager-chromium pushed a commit to szager-chromium/rr that referenced this issue Dec 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.