On MacOS, dtrace fails due to running as the wrong CPU architecture if the "cargo" binary's arch doesn't the arch of the program being profiled

[zbentley]

Environment

• M1 (arm/aarch64) MacOS 14.
• rustc 1.75 aarch64 stable toolchain and target.
• flamegraph-rs 0.5.1.
• SIP disabled.
• sudo-capable account.
• Rosetta 2 installed.
• cargo (the binary executable) compiled as non-native x86_64, running via Rosetta. E.g. you can reproduce the following:

> arch
arm64
> file $(which cargo)
/path/to/cargo: Mach-O 64-bit executable x86_64

That last part is important, and isn't as uncommon as you'd think due to e.g. subpar brew installations of rustup-init or other workstation provisioning automations that run an x86 binary somewhere in the stack before Cargo is installed.

It's also rarely a problem. Cargo-the-build-system still works totally fine compiling and cross compiling Rust programs for user-selected architectures/toolchains even if cargo-the-binary isn't built for the native hardware architecture. While not optimal, it doesn't even have major performance implications, since cargo-the-binary is usually delegating out to other (hopefully native-arch) binaries like rustc for most heavy lifting.

In some rare cases like this one, though, it does cause a problem:

Steps to reproduce

• Create a crate with a rust binary and a target architecture of aarch64. Doesn't matter what the binary does; hello world is fine.
• Do cargo flamegraph --root --bin $project

Expected behavior

Successful emission of a flamegraph SVG.

Observed behavior

> cargo flamegraph --root --bin untitled
    Finished release [optimized + debuginfo] target(s) in 0.00s
dtrace: invalid probe specifier profile-997 /pid == $target/ { @[ustack(100)] = count(); }: "/usr/lib/dtrace/darwin.d", line 26: syntax error near "uthread_t"
failed to sample program

Pathology

I'm writing a longer-form blog post about this which I'll link shortly. The short version is that when cargo is used as a subcommand runner, Rosetta's "architecture preference inheritance" kicks in. If a subcommand (e.g. the flamegraph binary) is spawned as a child of a non-native-architecture binary running under Rosetta, MacOS will try to use the non-native architecture for all child and grandchild, recursively processes.

In other words, if cargo is an x86 binary, even if flamegraph is an ARM binary (which causes MacOS to say "can you run as x86? No? Ok, ARM is fine then" and fall back), when flamegraph runs sudo, or when sudo runs dtrace, MacOS will keep trying to launch sub-sub-subcommands as x86.

This is problematic when dtrace is involved, since dtrace (and sudo) are distributed on MacOS as universal binaries, meaning that they can be run as either architecture. When its spawner expresses a preference to run as x86, dtrace will gladly do so, and will then attempt to trace an aarch64 user program.

This results in the less-than-informative "invalid probe specifier ... syntax error" failure in the dtrace stdlib import, which really means "x86 dtrace has no idea how to load symbols needed to trace an aarch64 binary".

Proposed resolution

flamegraph should probe the binary being traced and see what architecture(s) it is built for. If it's built for the native architecture (potentially among others), or if its only built for one architecture, it should override the architecture-preference "hints" with which sudo and dtrace are spawned in order to force the architecture-matching versions of those universal binaries to be used, even if cargo-the-binary has the wrong arch.

I'm drafting a PR to do this now, but I definitely am not invested in that fix. If this behavior is deemed rare enough as an edge case (though see the top of this report; I don't think it's too terribly rare), I could just update flamegraph's readme or help docs to explain the error and resolution steps (reinstall cargo as the right architecture) instead.

[zbentley]

I wrote some additional info about the discovery/identification of this issue on my blog, here in case other troubleshooting steps are useful for others.