Occasionally incorrect calling convention when working on PE files

[crimsonskylark]

Version and Platform (required):

• Binary Ninja Version: 5.3.8625-dev
• Edition: Non-Commercial
• OS: Manjaro Linux
• OS Version: 25.0.10
• CPU Architecture: x64

Bug Description:
BinaryNinja will sometimes assign the sysv calling convention when working on PE64 files.

Steps To Reproduce:
Please provide all steps required to reproduce the behavior:

1. Go to HvcallInitiateHypercall
2. Calling convention is set to sysv after analysis

Expected Behavior:
The win64 calling convention should always be used for PE64. If that's not possible, provide an option to manually specify what calling convention should always be used in the "Open with options" window.

Screenshots/Video Recording:

Binary:
metro horizon retries beautifully

Additional Information:
Unsure if this is a duplicate. I searched both open and closed issues but if it is feel free to close.

[monotonik-guy]

I think problem here

binaryninja-api/platform/windows-kernel/platform_windows_kernel.cpp

Line 77 in f52693d

cc = arch->GetCallingConventionByName("sysv");

You're reversing windows kernel PE file, and for some reason on it can use sysv. But I never actually meet sysv in windows kernel, maybe I wrong, but I think it always win64 or just pure for asm code.

[monotonik-guy]

@crimsonskylark you can try this:

1. Remove this line: it will completely remove this calling convention for kernel files, and only win64 will used for functions.
2. Build by yourself (what I recommend to do for safety reason).
   OR
   Download my build platform_windows_kernel.zip. (I'm using dev version btw, so Idk will it work on non-dev, but I think should) P.S. Idk, why output is much lighter, I just build with release. But it works anyway. Nvm, you're using linux, so need build by yourself.
3. Just put what get on previous step to plugins in root BinaryNinja folder.

[monotonik-guy]

Btw, after change for me it start infinitely analyzing on Indirect Branch Prediction Barrier longer than before, seems without sysv for BN hard to understand what happening here, and it fails on those functions

[Function] Skipping analysis for function at 0x1406b3947 : Exceeds 'analysis.limits.maxFunctionUpdateCount'. If possible, please report this issue.
[Function] Skipping analysis for function at 0x1406b3950 : Exceeds 'analysis.limits.maxFunctionUpdateCount'. If possible, please report this issue.
[Function] Skipping analysis for function at 0x1406b3935 : Exceeds 'analysis.limits.maxFunctionUpdateCount'. If possible, please report this issue.

1406a808e  4883c408           add     rsp, 0x8
1406a8092  e8eeffffff         call    sub_1406a8085  {sub_1406a8097}
{ Falls through into sub_1406a8097 }


1406a8097    int64_t sub_1406a8097()

1406a8097  4883c408           add     rsp, 0x8
1406a809b  e8eeffffff         call    sub_1406a808e  {sub_1406a80a0}
{ Falls through into sub_1406a80a0 }


1406a80a0    int64_t sub_1406a80a0()

1406a80a0  4883c408           add     rsp, 0x8
1406a80a4  e8eeffffff         call    sub_1406a8097  {sub_1406a80a9}
{ Falls through into sub_1406a80a9 }


1406a80a9    int64_t sub_1406a80a9()

1406a80a9  4883c408           add     rsp, 0x8
1406a80ad  e8eeffffff         call    sub_1406a80a0  {sub_1406a80b2}
{ Falls through into sub_1406a80b2 }


1406a80b2    int64_t sub_1406a80b2()

1406a80b2  4883c408           add     rsp, 0x8
1406a80b6  e8eeffffff         call    sub_1406a80a9  {sub_1406a80bb}
{ Falls through into sub_1406a80bb }


1406a80bb    int64_t sub_1406a80bb()

1406a80bb  4883c408           add     rsp, 0x8
1406a80bf  e8eeffffff         call    sub_1406a80b2  {sub_1406a80c4}
{ Falls through into sub_1406a80c4 }


1406a80c4    int64_t sub_1406a80c4()

1406a80c4  4883c408           add     rsp, 0x8
1406a80c8  e8eeffffff         call    sub_1406a80bb  {sub_1406a80cd}
{ Falls through into sub_1406a80cd }


1406a80cd    int64_t sub_1406a80cd()

1406a80cd  4883c408           add     rsp, 0x8
1406a80d1  e8eeffffff         call    sub_1406a80c4  {sub_1406a80d6}
{ Falls through into sub_1406a80d6 }


1406a80d6    int64_t sub_1406a80d6()

1406a80d6  4883c408           add     rsp, 0x8
1406a80da  e8eeffffff         call    sub_1406a80cd  {sub_1406a80df}
{ Falls through into sub_1406a80df }

[crimsonskylark]

@monotonik-guy Sorry for the delayed response. I appreciate you digging into the issue - I'll try your fix as soon as possible.

[plafosse]

Referencing this internal issue https://github.com/Vector35/binaryninja/issues/1708