5.4 regression: perf & flamegraph.pl do not work out-of-the-box anymore

[kit-ty-kate]

I'm currently using 5.4.0~alpha1 and i needed to do some profiling.
I followed the recommendation of the Profiling page on ocaml.org and from experience last time i used it, i knew that perf was a reasonable option.

However to my surprised this time things didn't work.
I tried to use flamegraph.pl to output a flamegraph as i need the extra UI to get to the information i need, but the program just failed as it expects UTF-8 strings (see brendangregg/FlameGraph#356).

Looking into the perf.data file and the output of the intermediate stackcollapse-perf.pl program, we can indeed find weird dangling characters such as 0xAD in the middle of otherwise fine text. Inconveniently these characters do not appear as a normal space in my terminal and vim didn't say anything either, which delayed my understanding of the problem.

Looking at the changelog i wasn't able to find anything related to perf, however fortunately i do follow the development of the compiler and i got a hunch that it might all be related to #13050. Reading the description and discussions it seems that perf needed some update, however it doesn't seem clear to me which version it was fixing and what exactly it fixes (was the issue esthetic or functional?)

• Is there a quick fix for this issue? Some sed or awk script somewhere maybe?
• If this happen to be indeed the expected breaking change, could a more prominent warning to perf/flamegraph.pl users be added to the changelog?

perf version 6.14.8-400.asahi.fc42.aarch64
OCaml version 5.4.0~alpha1 (works fine with 5.3.0)
Flamegraph version brendangregg/FlameGraph@41fee1f (as packaged by Fedora)

[kayceesrk]

CC @tmcgilchrist

[gasche]

I would like to understand better the nature of the regression here. #13050 changed some . into $ in some symbols. It is unclear why this would result in "weird dangling characters such as 0xAD" or non-unicode input. It may be that this PR is not in fact related to the observed change, or that there is a cascade of problems -- for example, maybe one of the tools in the chain interprets $foo as a variable to be interpolated?

[kit-ty-kate]

It may be that this PR is not in fact related to the observed change, or that there is a cascade of problems -- for example, maybe one of the tools in the chain interprets $foo as a variable to be interpolated?

Possibly yes. From @tmcgilchrist's suggestion, i gave inferno (a rust reimplementation of flamegraph.pl) a try and it worked perfectly. EDIT: see below

I'll update the upstream ticket with the new findings. If that's indeed the problem i would still suggest to add a warning in the changelog to flamegraph.pl users as a fix might not be available everywhere by OCaml 5.4.0 release day.

[kit-ty-kate]

Nevermind i spoke too fast.... i forgot the last run of perf was with OCaml 5.3.0 :/
Inferno does fail from a similar issue:

thread '<unnamed>' panicked at src/collapse/common.rs:489:64:
byte index 14 is not a char boundary; it is inside '�' (bytes 13..16) of `Stdlib.Buffer�d_substring_591`
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

thread '<unnamed>' panicked at src/collapse/common.rs:489:64:
byte index 14 is not a char boundary; it is inside '�' (bytes 13..16) of `Stdlib.Buffer�d_substring_591`

thread '<unnamed>' panicked at src/collapse/common.rs:489:64:
byte index 14 is not a char boundary; it is inside '�' (bytes 13..16) of `Stdlib.Buffer�d_substring_591`

thread '<unnamed>' panicked at src/collapse/common.rs:489:64:
byte index 14 is not a char boundary; it is inside '�' (bytes 13..16) of `Stdlib.Buffer�d_substring_591`

thread '<unnamed>' panicked at src/collapse/common.rs:489:64:
byte index 14 is not a char boundary; it is inside '�' (bytes 13..16) of `Stdlib.Buffer�d_substring_591`

thread '<unnamed>' panicked at src/collapse/common.rs:489:64:
byte index 14 is not a char boundary; it is inside '�' (bytes 13..16) of `Stdlib.Buffer�d_substring_591`

thread '<unnamed>' panicked at src/collapse/common.rs:489:64:
byte index 14 is not a char boundary; it is inside '�' (bytes 13..16) of `Stdlib.Buffer�d_substring_591`

thread '<unnamed>' panicked at src/collapse/common.rs:489:64:
byte index 14 is not a char boundary; it is inside '�' (bytes 13..16) of `Stdlib.Buffer�d_substring_591`

thread '<unnamed>' panicked at src/collapse/common.rs:489:64:
byte index 14 is not a char boundary; it is inside '�' (bytes 13..16) of `Stdlib.Buffer�d_substring_591`

thread '<unnamed>' panicked at src/collapse/common.rs:489:64:
byte index 14 is not a char boundary; it is inside '�' (bytes 13..16) of `Stdlib.Buffer�d_substring_591`

thread 'main' panicked at /builddir/build/BUILD/rust-inferno-0.12.2-build/inferno-0.12.2/src/collapse/common.rs:343:31:
called `Result::unwrap()` on an `Err` value: Any { .. }
[ERROR inferno::flamegraph] No stack counts found
Error: Custom { kind: InvalidData, error: "No stack counts found" }

[dbuenzli]

(Quick look)

I think this is expected. This comment mentioned that OCaml's demangling scheme would have to be updated by moving to use $ as the separator.

I guess you end up in this case which see it as byte escaped as hex, which is decoded into a byte with the next two bytes which turns out the be an invalid UTF-8 starter byte.

[yakobowski]

I agree with Daniel. We need to send a patch to the kernel, which takes into account the new demangling strategy. And possibly find how to recompile perf with our patch (while the changes land in mainstream kernels). Unfortunately, perf needs to be compiled with a specific version of the kernel, so the best we can probably do is to provide a script.

(Additionally, with my "C code verification" hat on, our current demangling code contains buffer overflows.)

[gasche]

Oops! I did remember the comment that the linux demangling would need updating, but I hadn't realized that the failure mode would be (arguably) so severe. Having perf working well seems important.

Maybe we should consider making the symbol separator a configure-time choice for now, with . instead of $ enabled by default, and try to get the demangling code updated in the kernel before we change the default? (This is a tradeoff between keeping current tools working, which is nice, and making the native debugger experience better in lldb, which also sounds nice.)

[yakobowski]

Maybe we should consider making the symbol separator a configure-time choice for now, with . instead of $ enabled by default, and try to get the demangling code updated in the kernel before we change the default?

That sounds very sensible to me. We're probably talking about (many) months (years?) before a fixed perf lands into mainstream distros.

[kit-ty-kate]

We're probably talking about (many) months (years?) before a fixed perf lands into mainstream distros.

without custom patches, probably yes, but distributions typically have custom patches on top of packages like that. If we were to ping the right people i bet this might possibly get fixed rather quickly.

[Octachron]

Another option would be to switch to $$ for module separator and $ for hexadecimal escaping to have better backward compatibility without having to maintain two versions of the demangling for years.

[copy]

Another option would be to switch to $$ for module separator and $ for hexadecimal escaping to have better backward compatibility without having to maintain two versions of the demangling for years.

I think this is a good idea. I agree that it would be better to find a solution that doesn't involve two incompatible mangling schemes. Unfortunately, $$ would cause the same problems as reported in this issue (for reference: https://github.com/torvalds/linux/blob/ee88bddf7f2f5d1f1da87dd7bedc734048b70e88/tools/perf/util/demangle-ocaml.c#L50-L64)

We could keep using . on Linux and use a $ or $2e on macOS/Windows. Once macOS's version of lldb catches up, we flip the mangling scheme to use ..

I did a quick test on Linux with gdb 16.3 and lldb 20.1.6:

• In lldb:
  • Breakpoints work: breakpoint set -F camlServer_main.code_begin
  • Disassemble works: disassemble -n camlServer_main.code_begin
• In gdb:
  • Breakpoints work: break camlServer_main.code_begin
  • Disassemble requires quoting: disassemble 'camlServer_main.code_begin'

[tmcgilchrist]

I haven't been able to reproduce your results @kit-ty-kate Do you still have the program and perf data file available?

I'm using Ubuntu 24.04 LTS (x86_64) with linux 6.8.0-60 and inferno 0.12.2. My sample program is a sudoku solver with some function names modified to trigger escaping.

let sólve' grid =
  search' (parse_grid grid)

let rec is_power_'' = function
  ...

let rec search' arr: square_value array =
  ...

$ nm -gD  sudoku_norvig.exe |less
...
000000000010b800 T camlSudoku_norvig$is_power_$$27$$27_751
000000000010c3a0 T camlSudoku_norvig$search$$27_978
000000000010c610 T camlSudoku_norvig$s$$c3$$b3lve$$27_988
000000000010c700 T camlSudoku_norvig$try_read_1192

Processing this using the rust inferno library I get names Sudoku_norvig$search$'_978, Sudoku_norvig$is_power_$'$'_751 and Sudoku_norvig$s$�$�lve$'_988 for the symbols with escapes. The resulting flame graph is attached.

flamegraph.svg

When using latin-1 characters the resulting name is not getting unescaped properly. That can be handled with a small modification to demangle-ocaml.c in perf.

[tmcgilchrist]

Something like this gets us back to properly escaped names with . separators:

ocaml_demangle_sym(const char *sym)
{
	char *result;
	int j = 0;
	int i;
	int len;

	if (!ocaml_is_mangled(sym)) {
		return NULL;
	}

	len = strlen(sym);

	/* the demangled symbol is always smaller than the mangled symbol */
	result = malloc(len + 1);
	if (!result)
		return NULL;

	/* skip "caml" prefix */
	i = caml_prefix_len;

	while (i < len) {
		if (sym[i] == '_' && sym[i + 1] == '_') {
			/* "__" -> "." */
			result[j++] = '.';
			i += 2;
		}
		else if (sym[i] == '$' && sym[i+1] == '$' && isxdigit(sym[i + 2]) && isxdigit(sym[i + 3])) {
			/* "$$xx" is a hex-encoded character */
			result[j++] = (hex(sym[i + 2]) << 4) | hex(sym[i + 3]);
			i += 4;
		}
		else if (sym[i] == '$') {
			/* "__" -> "." */
			result[j++] = '.';
			i += 1;
		}
		else {
			result[j++] = sym[i++];
		}
	}
	result[j] = '\0';

	return result;
}

To support the 4.14, 5.* and 5.4 escaping schemes in the same perf version requires a little more thought.

We could keep using . on Linux and use a $ or $2e on macOS/Windows. Once macOS's version of lldb catches up, we flip the mangling scheme to use ..

I don't expect the LLDB version that comes with XCode to catchup in future. Plus the current versions that come with XCode 15 and 16 should be supported.

[Octachron]

A correct possibility for non-ambiguous mangling across version:

• $XX for hexadecimal escaping
• $_ for module separation