feat: call-boundary ABI framework with strict clobber semantics#76
Merged
feat: call-boundary ABI framework with strict clobber semantics#76
Conversation
…nlining scaffolding Cross-ABI call contract (AbiCallContract.hpp): - AbiKind enum (x64_msvc, x86_cdecl/stdcall/fastcall, unknown) - CallModelMode: strict (default) clobbers volatile regs, compat preserves all - CallEffects: arg regs, return regs, volatile set, stack cleanup, memory effect - Pre-built descriptors for x64 MSVC and x86 calling conventions - Structured diagnostics at every call site ([call-abi] prefix) Call-site semantics (lift_call): - applyPostCallEffects: assigns RAX=result, clobbers volatile in strict mode - emittedExternalCall flag: skips Unflatten inlining when CreateCall emitted - Import thunk detection (FF 25 jmp [IAT]): auto-outlines DLL imports - shouldOutlineCall hook: extensible policy for inline/outline decisions Bug fixes: - parseArgs(nullptr) duplicated RDI (18 values for 16-type slots) — now 16 GPRs + memory ptr - Unknown calls in lift_call never assigned RAX = call result — now they do - callFunctionIR routed through applyPostCallEffects for consistency Speculative inlining (disabled by default, opt-in via maxCallInlineBudget): - Budget-limited call inlining with bail-out to CreateCall + ABI effects - Worklist trimming on bail-out restores pre-call continuation - Works mechanically but needs smarter trigger policy (see open issue) Tests: - call_abi_compat_preserves_volatile: R10 survives, RAX = result - call_abi_strict_clobbers_volatile: R10 = undef, RBX preserved, RAX = result - call_abi_default_is_strict: verifies strict is the default - All existing baseline (90+), semantic (23/23), micro (15) tests pass - VMP 3.8.1 target produces identical a+b+c deobfuscation
Documents the unsolved problem of distinguishing real function calls (library, CRT) from obfuscation call gadgets (VM handlers, push+ret). Current state: import thunks auto-detected, speculative budget mechanism built but disabled. Needs call-depth scoped policy or hybrid approach. See .github/SPECULATIVE_INLINE_ISSUE.md for full analysis and proposals.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Cross-ABI call-boundary framework that models what happens at non-inlineable
callinstructions (indirect calls, import thunks, outlined targets).Changes
New:
AbiCallContract.hppAbiKind: x64_msvc, x86_cdecl/stdcall/fastcall, unknownCallModelMode: strict (default, clobbers volatile) / compat (preserves all)CallEffects: arg regs, return regs, volatile set, stack cleanup, memory effectRegSet<Register>: fixed-capacity register set for ABI specs[call-abi]prefix at every call site)Modified:
lift_call(Semantics_ControlFlow.ipp)applyPostCallEffects: writes RAX = result, clobbers volatile regs in strict modeemittedExternalCallflag: skips Unflatten path when CreateCall was emittedFF 25→jmp [IAT]): auto-outlines DLL importsmaxCallInlineBudget)Bug fixes
parseArgs(nullptr)duplicated RDI (18 values for 16 type slots)lift_callnever assigned RAX = call resultcallFunctionIRnow routes throughapplyPostCallEffectsTests
call_abi_compat_preserves_volatile: R10 survives, RAX = resultcall_abi_strict_clobbers_volatile: R10 = undef, RBX preservedcall_abi_default_is_strict: verifies strict is the defaultVerification
a + b + cdeobfuscation (14K instructions, 1629 blocks)call r9(function pointer) binary: correct CreateCall + ABI effects + continuationOpen work
Speculative inlining policy for distinguishing VM handler calls from library calls — see #75