Got an error - "Error: PMU in use on target: 0x309 Active"

[rsg-ykkoo]

Environment

• CPUs : INTEL(R) XEON(R) GOLD 6548N *2
• OS : Ubuntu 22.04.5 LTS
• perfspect version : perfspect version 3.3.1_2025-02-07_2fa54460

Erros

• When i run perfspect with metrics option, i got a following error
  hostname ⢿ Error: PMU in use on target: 0x309 Active
• in the perfspect.log files
  time=2025-02-21T09:21:28.827+09:00 level=INFO msg="Starting up" app=perfspect version=3.3.1_2025-02-07_2fa54460 PID=8673 arguments="./perfspect metrics"
  time=2025-02-21T09:21:29.747+09:00 level=ERROR msg="failed to prepare target" target=dell-emr error="PMU in use on target: 0x309 Active"
  time=2025-02-21T09:21:29.747+09:00 level=ERROR msg="no targets were successfully prepared"

Does anyone have a good idea with this issue?

[harp-intel]

When PerfSpect starts up it checks if any of the PMU registers are in use. The instructions register is 0x309. If any other processes use this register, PerfSpect cannot continue.

[harp-intel]

@rsg-ykkoo Was this error persistent? Were you able to get past it?

[harp-intel]

Closing issue due to inactivity. If this issue comes back, please let us know.

[harp-intel]

I see this regularly but not consistently on AWS m7i.metal-48xl.

[hsane-dev]

Yes I am seeing this intermittantly as well, no clear indication if its a bug in perfspect or some other daemons/processes. I wonder if this MSR is being touched by any other AWS process?

[harp-intel]

@hsane-dev Do you see this issue on non-metal instances? Currently, I only see it on m7i.metal-48xl.

[harp-intel]

In the scenario where I'm seeing this error (m7i.metal-48xl), I've found that perf is taking longer to stop after being signaled, e.g., with Ctrl-C (SIGINT) than it takes on other systems. When I run the next test, perf hasn't fully stopped...so the test gets this "PMU in use..." error. In my test script, I added a check to determine if perf is still running before continuing.

[hsane-dev]

In my case, I was spawning both "metrics" and "telemetry" in the background. Unless I put them in the same Process group and send a SIGINT to the group PID, one lingers around causing a problem to the next collection, so seems resolved for now.
Theoretically, given the different things they do, perfspect should be able to launch report, telemetry and metrics simultaneously, but I understand the development overhead for a small feature.

[harp-intel]

Shutting down all the spawned processes (perf, sar, turbostat, etc.) cleanly when disconnected from the original shell, e.g., run in background, has been an ongoing challenge. I've addressed some of the scenarios, but it seems like there's more to do.

[hsane-dev]

Yes, its the numerous spawned processes. What may work is spawning these processes in the same process group (eg. setsid or set -m, for a new session as leader in bash) and then just terminate the leader, should kill all processes. Quick lookup shows "SysProcAttr = &syscall.SysProcAttr{Setpgid: true}" is the equivalent in Golang while spawning.
Wild idea but what if we create a knob, say "perfspect --kill metric/telemetry/all" as an easy user interface to kill the perfspect processes assuming the user sending the kill has the privileges.