X64: cmpxchg16b raises UC_ERR_INSN_INVALID

[domenukk]

While emulating the Linux Kernel in x64 bit mode, I encountered crashes with cmpxchg16b.
The following testcase ("lock cmpxchg16b xmmword ptr [r12]") crashes, for example:

#!/usr/bin/python
from unicorn import *
from unicorn.x86_const import *
from capstone import *
from capstone.x86 import *

def main():
    cs = Cs(CS_ARCH_X86, CS_MODE_64) # type: capstone.Cs
    uc = Uc(UC_ARCH_X86, UC_MODE_64) # type: unicorn.Uc

    # lock cmpxchg16b xmmword ptr [r12]
    cmpxchange16b = b"\xf0\x49\x0f\xc7\x0c\x24"
    insn_pos = 0xFFFFFFFF00b9a6000

    uc.mem_map(insn_pos, 4096)
    uc.mem_write(insn_pos, cmpxchange16b)
    try:
        uc.emu_start(insn_pos, len(cmpxchange16b), count=1)
    except Exception as ex:
        print("Error {}: ".format(ex), *(list(cs.disasm_lite(bytes(uc.mem_read(uc.reg_read(UC_X86_REG_RIP), 10)), 0))[0]))

    return 0

if __name__ == "__main__":
    main()

[domenukk]

Stepping through the latest version of unicorn, I can confirm the

void helper_cmpxchg16b(CPUX86State *env, target_ulong a0)

function is emitted and called during execution.
To not hit any segfaults, I had to add a random jmp to the testcase from above:

cmpxchange16b = b"\xf0\x49\x0f\xc7\x0c\x24" + b"\xff\x25\x00\x00\x00\x00"

Still investigating why I see/(saw?) illegal instruction in the first place and if it might still be an issue.

[aquynh]

This may be a bug in Qemu 2.1.2? You can look at the latest Qemu code to see if there is any changes, and port back the fix if possible.

[domenukk]

There were quite some changes for this opcode... My understanding of the qemu/unicorn loop is not deep enough to understand if this might fix a bug or just reordered things/introduced new functionality.
I opened a p/r with a backport to play around with

[domenukk]

Even without my PR this seems to be fixed for master.
Closing :)