v2.0.0b9

Speakeasy v2.0.0b9

This release changes what an API address is, records API arguments as named and typed objects, and fixes more than 200 bugs in the API handlers. Most of the handler fixes stop a Python exception that ended the run, correct a buffer size or return value rule, or fix the stack layout of a call on x86. Samples now run further, and their reports contain more events.

The report format is now version 5.0.0. See the breaking changes at the end.

API addresses are entries in mapped module images

An API address is now an entry inside a mapped PE image of the module that owns it, for example kernel32. The 0xfeedfXXX sentinel addresses outside guest memory are gone. The export table, the bound IAT slots, GetProcAddress, LdrGetProcedureAddress and MmGetSystemRoutineAddress all give the same address for a function. Shellcode that walks the PEB and the export tables gets the same pointers as code that uses the IAT. A guest can read, hook or patch the entry bytes. Speakeasy runs the Python handler when execution reaches the entry. (#350, #351)

The export tables of these images come from manifests of 121 x86 and 129 x64 Windows Server 2025 (10.0.26100) modules, including ordinal-only exports. An import by ordinal gets the same function as on that Windows build. Modules without a manifest use the signature catalog. (#350, #352)

Library users can call Speakeasy.get_api_symbols() to get an {address: "dll.name"} map of all API entries. (#351)

Named and typed API arguments

Each API event now records its arguments as objects with name, type, value and display. Handled calls get parameter names and enum and flag decoding from the signature database, as calls emulated from signatures already did in v2.0.0b7. About 700 of the 1046 handlers match a signature and get named arguments.

"api_name": "kernel32.CreateFileW",
"args": [
  {"name": "lpFileName", "type": "str", "value": 4160, "display": "C:\\missing.txt"},
  {"name": "dwDesiredAccess", "type": "flags", "value": 2147483648, "display": "GENERIC_READ"},
  {"name": "dwCreationDisposition", "type": "enum", "value": 3, "display": "OPEN_EXISTING"},
  ...
]

Several handlers showed their decoded value on the wrong parameter, for example the ZwCreateFile path on IoStatusBlock and the ZwWriteFile data on Length. These now show on the correct parameter. (#334, #335, #336)

Native DLL initialization and per-process module lists

Native DLLs that a guest image depends on now run their TLS callbacks and DllMain in load order, once per process. If DllMain fails during a runtime load, LdrLoadDll returns STATUS_DLL_INIT_FAILED. If a load fails with an emulator error, LoadLibrary returns NULL, and the run continues. Each process has its own PEB loader lists. The lists are now valid rings through their PEB_LDR_DATA heads. (#349, #352)

Execution limits

timeout and max_instructions now limit each run separately. When a limit is reached, the run records an error of type timeout or max_instructions, and the next queued run starts. The new max_total_time option (--max-total-time, default 0, disabled) limits the active execution time of all runs together. Time paused in the debugger does not count. (#349, #352)

GDB stub

When the sample exits without a fault, the stub now sends one T05 stop before the exit reply. A client can then read memory after ExitProcess, for example to dump an unpacked image (#330). A memory fault gives a SIGSEGV stop (T0b, then X0b). Before, it gave a clean exit (#331). Memory reads up to the advertised packet size succeed, so IDA's take_memory_snapshot no longer gets E01 (#332).

Breakpoints on API addresses now trigger. When the sample calls an API that Speakeasy cannot emulate, the debugger stops at the API entry before the call. You can add a hook, change PC or patch the entry, and then continue (#351, #352). A resume after a write watchpoint no longer repeats the write (#349). timeout also applies with a debugger attached, and paused time does not count (#352).

Loader and engine

PE section protections are applied in contiguous runs of pages. On a 7.2 MiB image, the load time decreases from 9.5 s to 0.6 s (#328). EXE inputs no longer get the container process image, which corrupted the first bytes of EXEs with ImageBase 0x400000. Config modules whose base is in use load at the next free base (#329). A malformed delay-import, export or import directory gives a warning, and the load continues with the valid entries. IAT slots keep their positions when an import name is malformed (#350).

Translated code is invalidated when a hook, a handler or the GDB stub writes guest memory, so patched code runs as written. The SEH replay stops when a handler returns EXCEPTION_CONTINUE_EXECUTION without a fix for the fault. Before, such a handler looped until the timeout. Ordinal imports of ws2_32 #177 to #181 no longer go to the wrong handler. Shellcode reports list the strings of the shellcode itself. Before, they listed the strings of the container image (#349).

On x64, executed code can read the kernel KUSER_SHARED_DATA page at 0xFFFFF78000000000. Before, these reads faulted because Unicorn truncates guest addresses to 52 bits. Speakeasy now maps a mirror of the page at the truncated address and writes the same data to all views of the page (#322, @warsang).

New API handlers

The version handlers GetFileVersionInfo, GetFileVersionInfoEx, GetFileVersionInfoSize, GetFileVersionInfoSizeEx and VerQueryValue report that the file has no version resource (#327, @tjnel). oleaut32.SysReAllocStringLen supports Free Pascal and Delphi WideString code (#333).

API handler fixes

The counts below are commits. Each fix has a test.

• kernel32 (61):
  • Buffer size queries and Windows length rules in WideCharToMultiByte, MultiByteToWideChar, ExpandEnvironmentStrings, GetEnvironmentVariable, GetLocaleInfo, LCMapString, GetFullPathName, GetModuleFileName, GetSystemDirectory, GetLongPathName, GetTempPath, GetComputerName, GetConsoleTitle, GetDateFormat and GetTimeFormat.
  • File I/O: SetFilePointer sign-extends the distance, _llseek uses its origin, TRUNCATE_EXISTING has its Windows value, MapViewOfFile maps the requested range, FindNextFile returns ERROR_NO_MORE_FILES, and GetFileAttributesEx writes the size.
  • Out-params that stayed uninitialized: GetSystemInfo, GetNativeSystemInfo, GlobalMemoryStatus, the volume queries, InitOnceBeginInitialize and RtlCaptureContext.
  • Python exceptions on unknown process or thread handles in WriteProcessMemory, ReadProcessMemory, CreateRemoteThread, VirtualProtectEx and OpenThread, and in OpenMutex, IsProcessorFeaturePresent, FlsGetValue2 and the volume enumeration.
  • Time: SystemTimeToFileTime converts its input (before, it wrote the current time, which broke kill-date checks), FileTimeToSystemTime counts weekdays from Sunday, and GetSystemTimePreciseAsFileTime no longer gives the year 7646.
  • VirtualQuery and VirtualProtect report the real page protection, TH32CS_SNAPALL snapshots keep all lists, Module32First and Module32Next fill szExePath, TLS indexes are shared across threads, and GetTempFileName follows the Windows naming and return value.
• ntoskrnl (38):
  • Calling conventions on x86: ObfReferenceObject and the fast mutex calls are fastcall, memmove, wcsnlen and mbstowcs are cdecl, 64-bit arguments use two slots (KeSetTimer, ExAllocatePool2, CmUnRegisterCallback), and _allshl uses EDX:EAX and CL.
  • Calls that ended the run now return an NTSTATUS: ZwOpenKey, ZwMapViewOfSection, ZwCreateEvent, ZwQuerySystemInformation, ZwQueryValueKey, RtlDecompressBuffer with XPRESS or malformed LZNT1 data, and ObReferenceObjectByName.
  • CRT string functions work on raw bytes and honor their counts: strchr, wcsncpy, _snprintf, _vsnprintf, mbstowcs and wcsnlen. RtlCompareMemory no longer counts one byte too many.
  • Pointer-width handles and CLIENT_ID on x64, NtCurrentProcess() and NtCurrentThread() in ObReferenceObjectByHandle and the context calls, and MmMapLockedPagesSpecifyCache returns the memory that the MDL describes.
• advapi32 (26) and the registry manager:
  • Registry values from the config now have their declared types, and RegQueryValueEx and RegGetValue return the data. Before, they returned no data.
  • RegEnumKeyEx returns subkey names, RegOpenKeyEx opens subkeys of an open handle, RegCreateKey returns the new handle, RegQueryInfoKey fills its out-params, and keys loaded from the config persist.
  • Accounts: LookupAccountSid, LookupAccountName and GetUserName follow the size query rules, and EqualSid compares the whole SID.
  • Crypto: CryptDeriveKey makes RC4 keys of the correct length, CryptGetHashParam returns the hash, CryptHashData accepts empty data, and CryptCreateHash fails cleanly for an unknown Algid.
• msvcrt (25):
  • String functions (strncpy, strncat, strcmp, _stricmp, strlen, strstr and others) work on raw bytes. Before, they decoded UTF-8 and dropped bytes. memcmp compares every byte, and tolower and toupper change only ASCII letters.
  • Count and truncation rules in strncpy, wcsncpy, strncat_s, _snprintf, _vsnprintf, vsprintf_s, wcstombs, mbstowcs_s and __stdio_common_vsprintf.
  • memset uses the low byte of the value, time writes 8 bytes on x64, _itoa, _ltoa and _itow use the radix, atoi parses only leading digits, and fseek takes a signed offset.
  • __getmainargs, __wgetmainargs, fputc and signal are cdecl, which fixes the x86 stack.
• user32 (14):
  • MSG has the SDK size, so GetMessage no longer writes past the caller's struct, and wvsprintf is stdcall.
  • DialogBoxParam, CreateWindowEx, FindWindow and RegisterClassEx accept atoms and resource IDs.
  • In wsprintfW, %s is a wide string. GetWindowText, CharUpperBuff, CharNext, OemToCharA and LoadString honor their lengths.
• shlwapi (14): StrStr, PathFindExtension and PathFindFileName return correct addresses for the W variants. PathAppend, PathIsRelative, PathRenameExtension, PathRemoveFileSpec and PathAddBackslash follow the Windows path rules. StrCmpI returns an ordering, and wnsprintf honors cchDest.
• ws2_32 (8): Unknown sockets fail with WSAENOTSOCK and no longer raise. WSAStartup fills WSADATA with the Windows layout, getaddrinfo accepts NULL arguments and allocates full structs, and accept writes the peer port and family. inet_ntoa, inet_addr(NULL) and gethostname with an exact-size buffer also work.
• wininet, winhttp, urlmon and dnsapi (12):
  • InternetCrackUrl and WinHttpCrackUrl split host, port and path, and HttpQueryInfo and WinHttpQueryHeaders answer status code queries.
  • POST and HEAD requests without a configured response get an empty body. Before, they ended the run.
  • DnsQuery answers DNS_TYPE_A, DNS names match without regard to case, and URLDownloadToCacheFile returns a failure HRESULT.
• ntdll (7): LdrAccessResource and LdrGetProcedureAddress no longer raise, and RtlComputeCrc32 starts from its initial value.
• KMDF, WSK, WFP and NDIS (wdfldr, netio, fwpkclnt, ndis, 14): WdfDriverOpenParametersRegistryKey no longer ends the run of a KMDF driver that reads its parameters. The USB pipe types, indexes and descriptor probes are fixed, WskReceiveFrom unpacks all of its arguments, and FwpmSubLayerDeleteByKey0 finds its sublayer. NDIS handles are pointer-sized on x64.
• COM (ole32, oleaut32, com_api, 7): CoCreateInstance and QueryInterface return REGDB_E_CLASSNOTREG or E_NOINTERFACE for classes and interfaces that Speakeasy does not emulate. Before, they returned S_OK. StringFromCLSID writes the GUID string, and SysAllocString allocates empty BSTRs.
• Other: shell32 (4: CommandLineToArgv follows the Windows rules, and SHGetFolderPath ignores CSIDL_FLAG_*), psapi (3), netapi32 and wkscli (5), ncrypt, bcrypt and crypt32 (6), secur32, rpcrt4 (UuidToStringA), iphlpapi (GetAdaptersInfo) and gdi32.

Breaking changes

Report format 5.0.0:

• args in API events is a list of {name, type, value, display} objects. Before, it was a list of strings.
• api_name names the module that owns the entry. For example, api-ms-win-crt-stdio-l1-1-0.__stdio_common_vfprintf is now msvcrt.__stdio_common_vfprintf.
• EXE inputs no longer list the container process image as a module.

Configuration and package:

• modules.module_directory_x86 and modules.module_directory_x64 default to null.
• The decoy images in speakeasy/winenv/decoys/ are removed. A module without a handler or a manifest gets an empty synthetic image.
• Exports of native DLLs loaded from the module directories run their own code. Handlers with the same name no longer intercept them.
• timeout and max_instructions apply to each run. The total time of an emulation can therefore reach max_runs × timeout. Use max_total_time to limit it.
• The signature archive is renamed to win32json_signatures.json.gz. A source checkout must run just gen-signatures again.

Handler and library API:

• Handlers take ctx: api.ApiContext = api.NO_CONTEXT, and they set display values with ctx.args["lpFileName"].display = .... Writes to argv no longer go into the report. API hooks get raw values in their params. Use emu.get_api_args() to get the values as the report shows them. See doc/api-handlers.md and doc/library.md.
• These emulator attributes and functions are removed: the import_table, impdata_queue and callbacks attributes, DecoyLoader, JitPeFile, and the imp_id parameter of load_pe(). Unused functions are removed from BinaryEmulator, WindowsEmulator, MemoryManager and _PeParser.
• load_pe() only parses the image and no longer sets the report input.
• A guest PE without relocations cannot be rebased, so its load fails when its base is in use.

Contributors

Thanks to @tjnel for the version.dll handlers, and to @warsang for the x64 KUSER_SHARED_DATA mirror.

Full Changelog: v2.0.0b8...v2.0.0b9