Incorrect parameter resolution

[fuzyll]

Originally posted by @yrp604 in #2558 (comment)

I think there are still problems with this. For example, here is notepad.exe:

14000cccc  int64_t DetermineFileTypeEncoding()

14000cce5      void var_468
14000cce5      int64_t rax_1 = __security_cookie ^ &var_468
14000ccf9      g_defaultEncoding // <- what is this?
14000cd0c      int32_t var_448 = 3
14000cd18      PWSTR lpFileName // <- this is presumably stack allocated?
14000cd18      HANDLE rax_2 = CreateFileW(lpFileName, dwDesiredAccess: 0x80000000, dwShareMode: FILE_SHARE_READ, lpSecurityAttributes: nullptr, dwCreationDisposition: OPEN_EXISTING, dwFlagsAndAttributes: FILE_ATTRIBUTE_NORMAL, hTemplateFile: nullptr) // <- right now it looks like were calling CreateFile on uninit stack contents?

This successfully created a variable named lpFileName, however this should have been an argument to DetermineFileTypeEncoding:

14000cccc  int64_t DetermineFileTypeEncoding()

14000cccc  48895c2410         mov     qword [rsp+0x10 {__saved_rbx}], rbx
14000ccd1  4889742418         mov     qword [rsp+0x18 {__saved_rsi}], rsi
14000ccd6  57                 push    rdi {__saved_rdi}
14000ccd7  4881ec60040000     sub     rsp, 0x460
14000ccde  488b050b770200     mov     rax, qword [rel __security_cookie]
14000cce5  4833c4             xor     rax, rsp {var_468}
14000cce8  4889842450040000   mov     qword [rsp+0x450 {var_18}], rax
14000ccf0  488364243000       and     qword [rsp+0x30 {var_438}], 0x0
14000ccf6  4533c9             xor     r9d, r9d  {0x0}
14000ccf9  8b1db1880200       mov     ebx, dword [rel g_defaultEncoding]
14000ccff  ba00000080         mov     edx, 0x80000000
14000cd04  c744242880000000   mov     dword [rsp+0x28 {var_440}], 0x80
14000cd0c  c744242003000000   mov     dword [rsp+0x20 {var_448}], 0x3
14000cd14  458d4101           lea     r8d, [r9+0x1]
14000cd18  48ff15c1d00100     call    qword [rel CreateFileW] ; <-- rcx is passed here

Looking at the asm, we can we see rcx isn't touched before the call to CreateFileW.

notepad.exe (2).txt