x86 x87 c1 becomes MLIL_UNIMPL/HLIL_UNIMPL when fnstsw reads status after fcos/fsin/fpatan/fst

[banteg]

Summary

I'm seeing a small, repeatable x86 x87 decompilation gap in Binary Ninja where
the instruction is decoded correctly, LLIL/MLIL still recognize the main x87
operation, but MLIL/HLIL introduce unimplemented for the c1 status bit when
the status word is read with fnstsw ax.

The issue reproduces in 7 bytes for fcos and fsin, and in 9 bytes for
fpatan.

Environment

• Binary Ninja version: 5.3.9245-dev Personal
• Build ID: 2729117674
• Host OS: macOS 15.6.1 (24G90)
• Analysis platform used for the raw repro blob: windows-x86

Minimal Positive Repros

fcos

fld1
fcos
fnstsw ax
ret

Bytes:

d9 e8 d9 ff df e0 c3

fsin

fld1
fsin
fnstsw ax
ret

Bytes:

d9 e8 d9 fe df e0 c3

fpatan

fld1
fld1
fpatan
fnstsw ax
ret

Bytes:

d9 e8 d9 e8 d9 f3 df e0 c3

Clean Controls

These stay clean in MLIL/HLIL on the same build:

fld1
fnstsw ax
ret

fld1
fcos
ret

fld1
fsin
ret

fld1
fld1
fpatan
ret

That points to the interaction being specifically "read x87 status after the
operation", not the operation by itself.

Binary Ninja Output

For the fcos repro:

Disassembly:

0x0 fld1
0x2 fcos
0x4 fnstsw ax
0x6 retn

MLIL:

0x2 result, c2 = __fcos(x87_r7_1)
0x2 c1 = unimplemented

HLIL:

0x2 bool c1 = unimplemented  {fcos }

For the fsin repro:

0x9 result, c2 = __fsin(x87_r7_1)
0x9 c1 = unimplemented

For the fpatan repro:

0x12 result = __fpatan(x87_r6, x87_r7_1)
0x12 c1 = unimplemented

This is why it looks like a core BN IL issue rather than a bad import or bad
typing issue:

• the instruction bytes are valid and decode correctly
• MLIL still recognizes __fcos, __fsin, and __fpatan
• the loss happens at the status-bit level, where c1 becomes
  unimplemented

Why This Matters

In a larger real-world target (crimsonland.exe, windows-x86), this same
class of x87 status handling correlates with much noisier HLIL in
player_update, where x87-heavy blocks degrade into many yellow
unimplemented expressions. I would file the tiny repro above first because it
is much easier to reason about than the full game function.

If useful, I also have a larger reduced artifact (player_update_full.obj)
containing only the raw bytes of that one function, but I would prefer to keep
the initial report focused on the 7-byte / 9-byte cases above.

Local Artifact Paths

• Source: minimal_x87_status_read.S
• Built ELF object: synth/minimal_x87_status_read.o
• Raw .text blob: synth/minimal_x87_status_read.bin

Function layout inside minimal_x87_status_read.bin:

• 0x00 repro_fcos_fnstsw
• 0x07 repro_fsin_fnstsw
• 0x0e repro_fpatan_fnstsw
• 0x17 repro_fst_fnstsw
• 0x1e control_fnstsw_only
• 0x23 control_fcos_only
• 0x28 control_fsin_only
• 0x2d control_fpatan_only
• 0x34 control_fst_only

Exact Validation Notes

I validated the raw blob inside the live BN GUI session by creating an in-memory
raw BinaryView, setting bv.platform = Platform['windows-x86'], adding user
functions at the offsets above, and calling update_analysis_and_wait().

Observed results:

• repro_fcos_fnstsw: mlil_unimpl=1, hlil_unimpl=1
• repro_fsin_fnstsw: mlil_unimpl=1, hlil_unimpl=1
• repro_fpatan_fnstsw: mlil_unimpl=1, hlil_unimpl=1
• repro_fst_fnstsw: mlil_unimpl=1, hlil_unimpl=1
• all controls: mlil_unimpl=0, hlil_unimpl=0

report_package.zip

[plafosse]

Thanks for detailed report and repro on this. I've fixed most of these issues but the FST one is giving me troubles and I'll need to take a deeper look into that one.