Perfcollect issue while executing perf script (kubernetes engine)

[sterchelen]

Hi,

First of all my workload is on Kubernetes engine. We run our pods on container optimized os from google. With this OS it's almost impossible to collect tracing events. So I created a node with ubuntu.
I can run perfcollect with no problem until the execution of the perf script statement which tries to output with cpu field. An error is thrown with the following description :
'cpu-clock' event do not have CPU attribute set.

• I run perfcollect as follow: ./perfcollect collect test -pid myPID

Does anyone has faced this issue ?

Thank you.

[jkotas]

cc @brianrob

[brianrob]

I have seen this once before, but it's not clear to me why it happens. It basically means that the CPU number is not available as part of the cpu-clock events.

If you want to work around this you can remove the cpu item from the calls to 'perf script' here: https://github.com/dotnet/corefx-tools/blob/master/src/performance/perfcollect/perfcollect#L1523.

I have filed an issue to see what can be done here: dotnet/corefx-tools#84

[tommcdon]

cc @sywhang

[sywhang]

This is showing up in more and more places, for example - dotnet/corefx-tools#92. This causes users to be unable to read their traces from PerfView because that error leads to perf.data.txt file not being populated in the trace zip file. I'll spend some time later this week to investigate why this might be happening.

[paraspatidar]

FYI , apart from kubernetes , same behavior is observed on Ubuntu VM on azure.
However on Ubuntu it happens only when we pass -pid in perfcollect as argument to specifically target perticular dotnet process.

for normal run where it collect for all CPUs (-a) , then it is successfully able to get CPU traces.
(however on kubernetes we have no other option but to use -pid as -a failed due to access issue)

when running that command manually :
/usr/bin/perf script -i perf.data.merged -f comm,pid,tid,cpu,time,event,ip,sym,dso,trace > perf.data.txt :
output : Samples for 'cpu-clock' event do not have CPU attribute set

OS :
NAME="Ubuntu"
VERSION="18.04.1 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.1 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

uname -r : 4.18.0-1011-azure

[sywhang]

I opened dotnet/corefx-tools#100 to track this instead of this issue, since it isn't really a runtime issue but more of the perfcollect script (corefx-tools is where that tool lives).

[sterchelen]

Is Microsoft planning to improve the performance metric ? Like go, integrating in the runtime some tools to extract EASILY cpu by instructions, memory usage (heap, garbage collection) etc.

…

On Fri, 5 Apr 2019, 01:05 Sung Yoon Whang, ***@***.***> wrote: I opened dotnet/corefx-tools#100 <dotnet/corefx-tools#100> to track this instead of this, since it isn't really a runtime issue but more of the perfcollect script (corefx-tools is where that tool lives). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/dotnet/coreclr/issues/18452#issuecomment-480094167>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AQfcaUNvJ3vXRxF2VhVUACL-hCupoFBgks5vdoVWgaJpZM4UllZF> .

[sywhang]

On Windows, there is already various ways that you can get do performance analysis such as ETW and PerfView.

To make this available on all platforms that we support on .NET Core, we are actively working on various runtime support such as EventPipe, runtime performance counters, etc. This project board (https://github.com/dotnet/coreclr/projects/5) tracks the work items that we are actively working on.

If you have specific questions about your scenario, feel free to reach out via issues on GitHub, or to my email (listed on my profile).

[lathiat]

I found this issue on Google when researching a similar error but in a different software suite.. sharing a few notes here in case someone comes back to this ticket:

I believe the reason this happens is that if you are on metal with hardware performance counters, perf uses the 'cycles' event that has the CPU number. If you are on a VM without hardware performance counters (which is most of the time, particularly in public clouds due to security reasons, I think some AWS instance types where you have dedicated hardware access might give you some performance counters even in a VM), then it uses the software 'cpu-clock' event which does not have this field.

I didn't get to figuring out why in some configurations using all-CPU versus per-process mode seems to result in the CPU field appearing or disappearing.

When using "perf script"'s "-F" field specifier you can specify different fields for "software", "hardware" and "trace" events. Unfortunately if you specify a format for hardware events and there are no such in the file then it errors out. So it seems we can't ask "perf script" to handle this for us and likely some kind of software detection is going to be required.

Hope it helps someone...