[unknown] functions

[do11]

Frequently I see in perf sript entries like this:

httpd 18294 [008] 205996.782023:   10101010 cpu-clock:
                  2b20e5 [unknown] (/opt/remi/php56/root/usr/lib64/httpd/modules/libphp5.so)
            2b5e1de3f698 [unknown] ([unknown])
            2b5e00000013 [unknown] ([unknown])
        4c20ec834853fd89 [unknown] ([unknown])

mysqld 18314 [003] 205996.822418:   10101010 cpu-clock:
                  39a7c4 prev_record_reads(st_position*, unsigned int, unsigned long long) (/usr/libexec/mysqld)
                  39c65c best_access_path(JOIN*, st_join_table*, unsigned long long, unsigned int, bool, double, st_position*, st_position*) (/usr/libexec/mysqld)
                  39e997 [unknown] (/usr/libexec/mysqld)
                  39ed0f [unknown] (/usr/libexec/mysqld)
                  39ed0f [unknown] (/usr/libexec/mysqld)
                  39ed0f [unknown] (/usr/libexec/mysqld)
...

Then flame graph is occupied with a lot of [unknown] functions, even though some of unknowns have binary associated with them. Could you modify stackcollapse-perf.pl if it sees [unknown] but there's binary to show binary name, it could be like [/path/bin] for example. Or add option to do such replacements.

[do11]

Btw, do you have any tips how to resolve these [unknown] ([unknown]) references?

[brendangregg]

Yes, maybe we should change stackcollapse-perf.pl as you suggest.

As for seeing [unknown] references: you've got to fix the profiler, whatever it is. In this case, Linux perf_events. Just as one example as a guideline:

On Java, I run the java process with -XX:+PreserveFramePointer (which costs a small amount of overhead), so that at least perf can walk stacks sensibly. At that point you still get [unknown], but at least the address aren't total garbage (which they can be if the app steps on the frame pointer). Then I run a java agent, perf-map-agent, (immediately after the profile), which dumps a /tmp/perf-PID.map file for perf to do symbol translations.

How this works for other apps is up to the app, but may be similar. You need a way for stack traces to work (eg, frame pointer-based, or DWARF-based), and, a way for perf to do symbol translations (some thing that writes the /tmp/perf-PID.map file).

[do11]

Thanks!

[liliang1213]

Hi,I've used '--perf-basic-prof' option when I profile nodejs,and I also get a file named '/tmp/perf-PID.map' under /tmp directory, but when I generate flame graph using 'perf script | ./stackcollapse-perf.pl | ./flamegraph.pl > MAIN.svg', it remains many 'unknown' label in my graph, how could that be?

perf script command returns like this:

[do11]

As I understood it, it could be just garbage values on the stack. Garbage in a sense that there is not valid frame pointers (probably optimized stack logic, for example, code compiled with -fomit-frame pointer).

[brendangregg]

I'm going to close this ticket because it's not a problem with Flame Graphs, which visualize the profiler output. If the profiler is emitting "[unknown]", then fix the profiler. It's system software, so ask your OS vendor.

I'd check if perf was complaining about file permissions, and fix that. Another issue has been buffering in v8, which was fixed recently -- look it up in v8 bug tracking, I don't have the URL on hand. I don't know when that fix will arrive in node, if it isn't there already. There are other issues that can cause "[unknown]" too, especially in JIT runtimes.