[Ubuntu] The perf_event_open syscall may not work, returning EPERM (permission denied)

[AlexTMjugador]

Description

As stated in the issue title, the mentioned system call may not work even if it is being executed by a privileged process after setting /proc/sys/kernel/perf_event_paranoid to a low enough value. Interestingly enough, the default value of that sysctl reads 4, which is not documented in the upstream kernel docs as a value that has any different meaning than 2.

I expect this system call to work after giving the process that executes it enough permissions, as GitHub-hosted runners are documented to run in an isolated VM, with its own kernel and virtualized hardware. I can't see how allowing greater performance counter access in an already potentially untrusted VM can have a meaningful impact in security.

Moreover, greater access to the performance counter kernel subsystem is so valuable for benchmarking native applications using metrics like instruction counts and branches, which are correlated to run-time performance, but much less sensitive to noise from the VM hypervisor, I/O and other processes. As far as I know, the most viable alternative to perf that can be used in GitHub Actions right now is Cachegrind, but Cachegrind virtualizes the CPU and is much slower.

There was some discussion about this shortcoming at the GitHub Community, but I'm not aware of any acknowledgment from any GitHub representative or satisfactory resolution for it. I hope that at least this issue brings some knowledgeable eyes to the situation, as there are definitely some people interested on it.

Virtual environments affected

• Ubuntu 18.04
• Ubuntu 20.04
• macOS 10.15
• macOS 11
• Windows Server 2016
• Windows Server 2019
• Windows Server 2022

Image version and build link

Image version: 20220123.1
Build link: https://github.com/AlexTMjugador/perftest/runs/4998008695

Is it regression?

No response

Expected behavior

The perf subsystem should just work, maybe barring some hardware-specific things that are not feasible to virtualize.

Actual behavior

The perf subsystem does not work. perf stat ls shows that some performance counters are not available, but they work on my Linux box for privileged processes, and unprivileged processes after changing the perf_event_paranoid sysctl.

Repro steps

1. Write any application that does the syscall with at least some combination of parameters. In my case, I blatantly copied and pasted a Rust example by @matklad, shared on the linked GitHub Community post, to my own repo, to be able to run it on GitHub Actions. The Rust source code that does the troublesome syscall is as follows: https://docs.rs/perf-event/0.4.7/src/perf_event/lib.rs.html#591-620
2. Build and run it, and watch it fail on a Ubuntu GitHub Actions runner, no matter what.

[dibir-magomedsaygitov]

Hello @AlexTMjugador. Thank for your report. We will take a look.

[mikhailkoliada]

@AlexTMjugador Hello!

Github-hosted runner images (Ubuntu and Windows specifically) are generated on the top of Ubuntu/Windows images from Azure marketplace, perf is a part of the linux kernel which we do not modify at all, in practice this means that if you try the same perf commands on a clean Ubuntu VM in Azure the result is going to be literally the same. Please file a support request against Azure directly.

I am going to close this ticket now, but feel free to reach us out if you have any questions left.

[AlexTMjugador]

Thank you for the help and explanation!

However, I haven't bought any support plan on Azure, and when I try to create a support ticket I can only do so for billing or subscription management matters, not for technical support. The next available option for me seems to be community support, and I've searched the Q&A and documentation for topics related to Linux performance counters availability without success.

I guess I could ask a question there, but would you suggest doing so? If so, may I refer to this GitHub issue or some other piece of information in that question?

[mikhailkoliada]

@AlexTMjugador oh I've completely forgotten that the support is primarily pay-based.

Have you tried glancing these boards?
I believe perf is quite popular tooling and you may find advice there.

[AlexTMjugador]

Thank you, I'll see what I can do over there 😄