[CoreCLR] Fix macOS jitdump discovery by Samply

[EgorBo]

On macOS, the runtime-wide __DARWIN_NON_CANCELABLE setting maps open to open$NOCANCEL. Samply discovers jitdump files by interposing regular open, so it stopped seeing CoreCLR jitdump output after #117330.

Bind only jitdump creation to plain _open on macOS. Linux and Apple mobile are unchanged.

Example usage of Samply:

This restores mixed native/managed stacks and annotated managed assembly, for example:

uint64 ManagedPipeline::ManagedHash(uint32,uint32)

   self%  samples  offset   instruction
     0.00        0  +0x0028  mov w3, #0x30
>>   1.72        2  +0x002c  eor x0, x0, x0, lsr #29
     0.00        0  +0x0030  mul x0, x0, x1
>>   6.03        7  +0x0034  ror x0, x0, #0x2f
     0.00        0  +0x0038  add x0, x0, w2
     0.00        0  +0x003c  mov w4, #0x79b9
     0.00        0  +0x0040  movk w4, #0x9e37, lsl #16
>>   7.76        9  +0x0044  add w2, w2, w4
>>  81.90       95  +0x0048  sub w3, w3, #0x1
     0.00        0  +0x004c  cbnz w3, $-0x20
>>   2.59        3  +0x0050  ldp x29, x30, [sp], #0x10
     0.00        0  +0x0054  ret

Validated with a Release CoreCLR build and Samply 0.13.1 on macOS arm64.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR adjusts CoreCLR’s jitdump file creation path on macOS to ensure the jitdump file is opened via the plain open symbol (rather than the $NOCANCEL-aliased variant), improving external profiler/tool discovery of jitdump output on macOS without affecting Linux or Apple mobile targets.

Changes:

• Introduces a JitDumpOpen binding on TARGET_OSX that links directly to the _open symbol.
• Switches jitdump file creation to use JitDumpOpen(...) instead of open(...).

[EgorBo]

@jkotas @janvorli @davidwrighton opinions? I assume Perfmap is effectively unused today for macos and this PR at least makes it work with samply.

My end-goal is to use samply on egorbot for -profiler on macos platforms.

[jkotas]

Should this be fixed in Samply instead? This change is introducing a subtle bug.

[hamarb123]

NOCANCEL is not even the only possible suffix: https://github.com/apple-oss-distributions/xnu/blob/f6217f891ac0bb64f3d375211650a4c1ff8ca1ea/bsd/sys/cdefs.h#L878-L909 - no doubt apple will make more in the future too - and on intel macOS, in a default setup these days, you will actually get $INODE64 on default calls to whole host of relevant APIs iirc (not including open, but including a bunch of others) today (and since macOS 10.7 or something like that iirc), because they wanted to break the behaviour w/o breaking existing stuff; so yeah, this should definitely be handled by samply imo, by ignoring the suffix, or having a list of recognised suffixes, or similar.

Definitely should not be on .NET to potentially leak memory to make it work, imo (although, maybe it's not the end of the world for a one-off allocation as a stopgap solution, but imo ideally it is not like this long-term).

[EgorBo]

Ok, I'll see if I can fix that on the Samply side