Skip to content

Releases: sp00nznet/xboxrecomp

v0.16.0 — "Small Print"

Choose a tag to compare

@sp00nznet sp00nznet released this 21 Oct 01:45

Four PRs, every one about a few bytes the tools read past or read too wide.
A push cx moved the stack by four bytes where the CPU moves it by two. An
MSVC byte index table, 00 02 02 00, was read as one more dword arm of the
switch in front of it. Functions behind nop padding went unnoticed because
only int3 counted as padding. And a save file opened by its bare name,
Data.ssx, lost the directory handle that said where it lived. Each was
small, and each stopped a game: Jet Set Radio Future's attract demo, SSX 3's
saves on Linux, and Panzer Dragoon Orta's first boot.

Contributed: disassembler

  • Vtable methods that never reach a ret inside their own gap are found.
    The closed-body proof refused 103 of JSRF's methods. They were guarded vcall
    tails (test eax, eax ; je done ; … ; jmp [eax+N]), dispatchers whose
    switch arms are other methods, and error paths that leave by a direct jmp
    to a helper nobody had found yet. A body may now close through an indirect
    tail via memory, through a proven entry, or through a direct branch whose
    destination is proven the same way in its own gap, eight levels deep. A
    byte index table is never taken as an entry: v0.15.0 had accepted eight
    such data words as functions. A measured jump table also stops in front of
    the byte table packed after it. On JSRF, 111 of the 113 methods are now
    entries, up from 10, and the attract demo's three are all found —
    @nworkers (#236)

Contributed: kernel

  • The POSIX file backend opens names relative to a RootDirectory
    handle
    , as the Win32 backend already did. XAPI's save code opens the save
    directory, then each file in it by its bare name. On Linux, overwriting an
    SSX 3 save failed with "Unable to save" (STATUS_OBJECT_PATH_NOT_FOUND).
    NtCreateFile, NtOpenFile, NtDeleteFile and
    NtQueryFullAttributesFile all take the join. A new cross-platform test,
    tests/kernel_file_relative, covers it —
    @bertibertig (#237)

Maintainer: found by Panzer Dragoon Orta

  • 16-bit push and pop move esp by 2. pop cx emitted POP32(esp, cx), which did not compile. push cx compiled but pushed four bytes,
    leaving the stack two bytes off. A pop to memory now writes after the
    increment, as x86 does —
    @sp00nznet (#238)
  • Functions past nop padding are gap functions. _pass_cc_boundaries
    only knew int3 padding, so Orta's COM forwarders, reached only through
    vtables, went unfound, and the first one called at boot was an unresolved
    indirect call. The gap pass now sweeps the addresses past padding first,
    then the ones straight after a ret. Doing both in one sweep split a
    forwarder's own error return off as a function. On Orta: 17,764 → 17,933
    functions, unresolved call stubs 71 → 41, and it boots to the SEGA and
    Smilebit logos —
    @sp00nznet (#239)

Testing

  • @dplewis ran a fresh macOS install of
    v0.15.0: Burnout 3 boots through the EA, Criterion and RenderWare logos to
    its title screen, with the keyboard working (#19).
  • tests/kernel_file_relative and tests/kernel_file_status build and pass
    under MSVC x64 and on Linux (WSL, GCC 14).
  • Half-Life 2, regenerated and rebuilt on this release. It finds 51,114
    functions (51,104 before), lifts every one, reaches its title screen, and
    runs 90 s with no faults and no unresolved indirect calls. MSVC pads with
    int3, so #239 finds little there, as expected.
  • A clean run is now 936 passed / 0 skipped, up from 916, plus 6,368
    conformance vectors with no mismatches on an AMD host.

Known issue

  • Burnout 3 (sp00nznet/burnout3) built against main stops before its first
    draw. The break is somewhere in afeb69f..fb09636, before v0.15.0, and is
    being bisected; burnout3 pins afeb69f until then (#240).

v0.15.0 — "Second Reading"

Choose a tag to compare

@sp00nznet sp00nznet released this 19 Oct 14:45

Twenty-four PRs from six people, plus one bug report. Much of what is fixed
here was caught because contributors read and ran each other's work before it
merged. @apexJCL tested
@koenhendriks's SSX 3 PRs on Burnout 3 and
BLiNX 2 and found three things. A DVD region had landed in MiscFlags. With
vsync off the present spun a core at 8,500 loops a second. A clipped vertex
could come out with w ≤ 0. @vyanhursky's
Def Jam test notes found an interrupt snapshot that dropped
g_eflags. A
Darkwatch bring-up sent a measured report, a patch and a watchdog dump. Even
the test oracle needed checking: the new x87 trig model matches the Xbox's
Pentium III, and an AMD test machine disagreed with it, correctly. Everything
below was looked at twice.

Contributed: lifter, recompiler and disassembler

  • Lifter accuracy and discovery past switch tables. The changes:

    • movsx r32, bp/sp used to zero-extend; it now sign-extends.
    • pushfd/popfd go through a g_eflags image, with OF and PF rebuilt
      from cmp, sub, add and the logic ops. cmp al, bl; pushfd with
      -128 - 1 had pushed OF clear.
    • cpuid answers as the P3.
    • x87 trig reduces with the 66-bit pi the hardware uses.
    • pop esp leaves esp at the popped value.
    • MSVC offset frames own their frame.
    • RECOMP_STUB_LOG=1 names each unresolved stub the first time it runs.
    • Code sections are decoded to their virtual end.
    • Functions after an embedded switch table are found.

    @apexJCL (#186)

  • An unresolved target inside a function enters that function.
    --entry-stubs re-emits the containing function from the target instead of
    returning at once. It's opt-in, because each target gets a full copy.
    Shared epilogues get a stub that pops their registers —
    @koenhendriks (#207)

  • XDK register constants no longer cut .text functions. With
    --skip-library-immediates, an immediate in a library section (D3D's NV2A
    constants, which can equal .text addresses) never starts a function.
    Found on NFSU1, where it split a switch —
    @kaskabayev (#234)

Contributed: kernel, interrupts and timing

  • Interrupts, DPCs and pacing, every part behind a switch that is off by
    default:

    • RECOMP_DISPATCH_GATE=1: a one-CPU DISPATCH gate, with DPCs at the raise
      and interrupts at safe points.
    • RECOMP_VBLANK=paced: a 60 Hz vblank schedule with an ack thread.
    • RECOMP_FLIP_PACING=1: flips held to the guest's vblank.
    • RECOMP_GUEST_CPUS=<n>: guest threads pinned to fewer cores.
    • The lifter's --back-edge-yield: loop checks for the guest CPU
      scheduler.

    RECOMP_GUEST_LOCK=1 used to be a plain critical section and is now the
    quantum/hand-off scheduler. With nothing set, the runtime and the generated
    code behave as in v0.14.0 —
    @apexJCL (#192)

  • The host time zone reaches the title. XC_MAX_OS and the single
    time-zone queries return the host's bias, names and daylight dates instead
    of zeroes, so saves stop being stamped UTC —
    @koenhendriks (#216)

  • Partition1\CACHE goes to <save_dir>/TitleCache. It used to go into
    the game directory: every run wrote into the extracted game files, and all
    save directories shared one cache —
    @apexJCL (#206)

Contributed: graphics, video and audio

  • Pushbuffer executor:

    • Attributes with no array read their current value. SSX 3's front end had
      collapsed onto the viewport offset.
    • SET_FOG_PLANE and SET_EYE_POSITION reach c[57] and c[56].
    • The oD0/oD1 colour outputs are clamped.
    • Mirror and border addressing are implemented.

    @koenhendriks (#210)

  • Clipping happens in clip space, and at the near plane. It used to
    happen after the viewport scale, which is a constant the title owns —
    @koenhendriks (#211)

  • Render to texture and CLEAR_RECT. Swizzled colour surfaces are
    written in Morton order, so SSX 3's shadow maps sample correctly. A clear
    covers the clear rect alone, as xemu does —
    @koenhendriks (#212)

  • A scalable Windows window, fullscreen, and an opt-in D3D11 presenter
    (RECOMP_D3D_PRESENT=1, RECOMP_PRESENT_VSYNC). Closing the window exits
    through the exit hook —
    @koenhendriks (#215)

  • PVIDEO overlay scanout. Bink's Xbox player draws through the NV2A
    overlay, so movies never appeared. The YUY2 overlay is now drawn over the
    framebuffer at its scale and colour key, and STOP clears the pending banks
    the title waits on —
    @MattSG (#220)

  • Voice processor fixes:

    • ADPCM reads used to overrun the decode buffers; they now stay inside
      them.
    • VOICE_LOCK locks one voice, instead of waiting for a whole frame of
      mixing.
    • Volume changes ramp across a frame.
    • HRTF is on, as in xemu; RECOMP_APU_HRTF=0 turns it off.

    @MattSG (#217)

Contributed: input and platform

  • Portable controller mapping rules:

    • named bindings and a text format for them;
    • remapping;
    • radial deadzones and trigger thresholds;
    • merge rules.

    They're built on #191's key table, and RECOMP_KEYBOARD is the first
    adopter, byte for byte the same as v0.14.0 —
    @vyanhursky (#203)

  • Four root-hub ports, and the first pad on controller port 1. The pad
    used to arrive as player 3, so a title that reads only slot 0 saw no
    controller. RECOMP_USB_PORT=0 / RECOMP_USB_NDP=2 bring back the old
    layout —
    @koenhendriks (#214)

  • Rumble. The XID output report is forwarded to the host pad;
    RECOMP_RUMBLE=0 turns it off —
    @koenhendriks (#213)

  • mingw-built exes start on stock Windows. They link xinput1_4 instead
    of the redistributable's xinput1_3 —
    @apexJCL (#221)

  • NOTICE lists the xemu-derived NV2A core, NV2A state and APU API
    headers
    — @kaskabayev (#233)

From the Darkwatch report
(@ranzo312-cyber, #127)

  • In-thread guest callbacks give the caller its registers back. APCs,
    inline DPCs, NtUserIoApcDispatcher and KeSynchronizeExecution run guest
    code inside the kernel call. The loader resumed with an APC's RDTSC value
    in esi and faulted.
  • RECOMP_CONTIG_AVOID_IMAGE=1 keeps contiguous blocks out of the image's
    pages.
    In this runtime one physical number can name both the window and
    the image, and the pad never enumerated because XPP's descriptors resolved
    to the wrong one. It's opt-in, because it moves every block (#235).
  • lahf after add emitted _cf undeclared. #186 had the same fix;
    the report's tests are kept.

Found by rebuilding real titles (maintainer)

  • Three v0.14.0 regressions, found by rebuilding Half-Life 2, which no
    test had caught:

    • lock xadd didn't compile.
    • Code after int3 padding had to probe as code, so mov eax, N; call __chkstk frames vanished and the game segfaulted at boot.
    • This-adjusting vcall thunks vanished.

    HL2 now reaches its title screen (#224).

  • Fallout: Brotherhood of Steel:

    • File I/O APCs left esp 12 bytes high, and can now be deferred to the next
      alertable wait (#226).
    • Code-section gaps are scanned for function-pointer tables, skipping
      switch tables; D3D's render-state handlers had no body (#227).
    • A batch's triangles rasterise together on the worker pool. The first
      level went from 5.5 fps, with the workers idle (#228).
    • The per-kick DMA_PUT line only prints under RECOMP_NV2A_TRACE (#230).
    • conformance --xbe builds on titles with untranslated instructions
      (#229).

Follow-ups made while merging

  • Conformance and fpdiff on AMD hosts. Both use the host CPU as the
    oracle, and only Intel's x87 reduces like the P3. On AMD the trig rows
    with pi/4 < |x| < 2^63 are dropped; tests/x87_trig checks them against
    the model everywhere.
  • g_eflags in the IRQ snapshot, the point #186 and #192 met at.
    guest_cpu and vblank_ack raise the timer resolution, because at
    Windows' default 15.6 ms tick they failed natively.
  • XC_MAX_OS's layout. It's 0x60 bytes, DVD region at 0x5C, and a short
    buffer gets STATUS_BUFFER_TOO_SMALL.
  • Present pacing. It's paced only when vsync is on.
  • A clipped vertex sits exactly on its plane.
  • CLEAR_RECT needs both halves, and swizzled writes stay inside
    1<<log.
  • ADPCM buffers hold every block count the 5-bit field can name, so a
    stream voice is never refused and left hanging.
  • The input layer: both Shifts are released on VK_SHIFT.
  • xbox_video links xbox_nv2a. #220's overlay scanout calls into
    nv2a_mmio_hook.c, and a game that links the libraries one by one (HL2)
    failed to link. The Half-Life 2 rebuild caught it.

Held for the next round, with reviews posted:

  • #222, Black bring-up 2. Its shadow and critical-section caches tear.
  • #231, the pipelined rasteriser. It needs a rebase over the NV2A PRs above.
  • #197, the flags carried across calls. It is still waiting on its
    size-cost rework.
  • #209, timing. It is ready to rebase now that #192 is in.

Testing

  • Every tests/* project builds and passes under MSVC except the POSIX-only
    ones and d3d8_smoke, as before, and #221's `input...
Read more

v0.14.0 — "Other Machines"

Choose a tag to compare

@sp00nznet sp00nznet released this 16 Oct 08:53

Twenty-two contributed PRs from six people, and almost none of it was found on the
machines or games this toolkit was built on. The runtime now runs on macOS
and Linux, arm64 included, because a contributor ran BLiNX 2 and Burnout 3 on a
Mac and a Steam Deck. On Jet Set Radio Future,
lock xadd had no flags, so
every COM
Release freed its object twice. Dead or Alive 3 hands its engine
callbacks the disassembler never saw. A Def Jam port on Apple Silicon checked
the Darwin mapping fix against its own regression suite before approving it.
On Black, DirectSound waited forever for an interrupt the APU never raised.
Most of these bugs live in shared code and would have reached any title. They
were found on other people's machines.

Contributed: lifter and recompiler

  • lock xadd had no flags, so InterlockedDecrement; jne never branched,
    every COM Release destroyed its object, and the double free corrupted the
    heap's free list. FCMOVcc was a comment, so MSVC's float min/max never
    moved. MXCSR is now modelled: stmxcsr stored nothing, and
    cvtss2si/cvtsd2si now round by its RC field instead of the host's —
    @nworkers (#198)
  • x87 and SSE forms that lifted to the wrong value: sahf, the fcomi
    family, tbyte loads and stores, ficom, the trig 2^63 limit, partial fprem,
    fscale/f2xm1/fyl2xp1, cvtss2si rounding, rcpss/rsqrtss estimates
    and SSE2 scalar double lanes. All of them were found with tools/recomp/fpdiff.py,
    which runs a snippet lifted and on the CPU and compares the two —
    @NoRain211 (#183)
  • COMISD/UCOMISD read the float lane, so 1.0 and 2.0 as doubles could
    compare equal —
    @vyanhursky (#178)
  • Parity setcc/cmovcc readers, including the pe/po spellings, read
    their bit after a known flag producer —
    @vyanhursky (#180)
  • FPREM/FPREM1 report completion: C2 is clear after a full reduction, so
    a CRT fmod loop that polls it terminates. Also covered by #183's
    recomp_fprem; its two conformance cases are kept —
    @vyanhursky (#177)
  • in/out lift to runtime port I/O. They were RECOMP_UNIMPL, so in al, dx
    left AL unchanged. The video encoder's field pin at 0x80C0, which D3D's vblank
    ISR polls, is modelled; every other port reads 0 and logs once. Takes effect
    on regeneration —
    @apexJCL (#194)
  • neg negates in unsigned arithmetic. (uint32_t)(-(int32_t)v) is
    undefined for 0x80000000, and clang -O2 miscompiled it. Found when #179 ran
    the fixtures at -O2.

Contributed: function discovery

  • Game callbacks static discovery missed. DOA3's callbacks sit outside every
    bounded table, or in gaps of unlisted code. They are now recovered from
    reachable immediates and tail branches, and a weak immediate must start on a
    code boundary. That rule rejects every false start review found on
    Wreckless —
    @NoRain211 (#176)
  • Callbacks and tails the disassembler missed. A closed-CFG proof replaces
    the instruction budget, late table aliases join the tail closure, and
    out-of-phase sweep bytes realign on seeds. Each fix cleared a stop the runtime
    hit on DOA3. On Wreckless it also drops 129 junk starts —
    @NoRain211 (#182)
  • Branches out of alias bodies into gaps: a vtable-only dispatcher branched
    past its own ret into bytes nothing claimed, and each call came back 12
    bytes short on the stack —
    @nworkers (#199)

Contributed: macOS, Linux and arm64 hosts

  • Devices on POSIX hosts. Trapped MMIO on arm64 is decoded from the faulting
    instruction, and one fault dispatcher serves the APU, AC'97 and OHCI. There is
    an SDL2 framebuffer window and audio device (RECOMP_FB_WINDOW), and one
    keyboard table and Xbox map for every host. On Windows the keyboard map
    changes
    : X/Y move to C/V and the left stick is on WASD —
    @apexJCL (#191)
  • Memory layout and POSIX fixes. The toolkit region, stack and heap sit above
    the highest section, so a large image's .data is no longer overlapped, and
    every section loads. The contiguous arena became a page allocator that honours
    MmAllocateContiguousMemoryEx's range: D3D's request for the low 16 MB
    failed once the bump pointer passed it. RECOMP_WIN_PLACEHOLDER (opt-in)
    reserves the 4 GB guest window as one placeholder, which keeps the aperture
    mappable under Proton —
    @apexJCL (#189)
  • A failed fixed view on Darwin leaked its placeholder, and every later view
    at that address was refused —
    @apexJCL (#185), tested on a Def Jam port by
    @vyanhursky
  • tests/proton_run.sh runs mingw-built Windows tests under Proton as
    ctest's emulator —
    @apexJCL (#187)

Contributed: kernel and APU

  • Kernel objects and files. Worker threads get a guest thread object, with
    TLS sized from TlsDataSize. The other changes:

    • FATX overwrite and rename;
    • partition images on POSIX, without which XAPI's utility-drive mount fails;
    • NtFreeVirtualMemory answers in guest terms;
    • a missing game file is reported once by name (RECOMP_MISSING_FILES);
    • POSIX NtClose drops a handle's directory search, which
      @vyanhursky also found (#201).

    @apexJCL (#190)

  • Guest-built events in multiple-object waits. A KEVENT the title built by
    writing its header now resolves in KeWaitForMultipleObjects, as it already
    did for a single wait —
    @vyanhursky (#181)

  • The APU. Physical addresses above 64 MB were folded into the low 64 MB.
    Voices are resampled by pitch: 44.1 kHz material had played 8.8% fast. The
    trapped front end takes one trap at a time, output is paced by the device,
    and FETINTSTS is latched once per trap. The DSP doorbell ack is now found
    from GPSADDR by default
    (RECOMP_APU_DSP_ACK=auto); an explicit address,
    or 0/off, still overrides —
    @apexJCL (#193)

  • Diagnostics for audio reports, all off by default:

    • RECOMP_AUDIO_WAV writes exactly what the device is given, and the file
      stays valid however the run ends;
    • RECOMP_AUDIO_HOST lines up host starvation with that file;
    • RECOMP_APU_TRACE adds per-voice statistics, RECOMP_APU_SOLO mixes one
      voice, and RECOMP_APU_RING finds a streamed ring replaying a stale lap.

    @apexJCL (#195)

  • The sequencer's sample-limit interrupt. DirectSound programs SSL and
    waits for ISTS bit 3. Without it, Black re-ran its audio setup forever.
    XGSCNT is now a 48 kHz sample counter, not the host clock divided by 100 —
    @LeandroSSB (#205)

Contributed: graphics

  • The pushbuffer executor. It adds fixed-function transform and lighting,
    anti-aliased surfaces, render back ends, bounds on every vertex fetch and
    surface write, and statics allocated on first use. The DMA-style walk is
    opt-in (RECOMP_PB_WALK=dma): with it, D3D's fence wait sleeps on a
    software-method interrupt the runtime doesn't raise yet —
    @BearddOddity (#162)
  • oPos.w == 0 vertices were dropped as behind the eye. A program that writes
    screen-space x/y/z itself leaves w at 0, so menus and loading screens drew
    nothing on the executor path. They now draw with w = 1 —
    @apexJCL (#188). Found independently on
    Black by @soldieg (#127) and
    @LeandroSSB (#205), whose frontend went
    from 0 pixels to its boot scene at 42 fps.
  • Vertex-constant tracing under RECOMP_VSH_DUMP shows every constant
    load, the values it completes and the writes dropped past the end of the
    file —
    @LeandroSSB (#205)

Contributed: input

  • A mouse and keyboard pad (RECOMP_MKB, Windows, off by default):
    raw-input mouse movement becomes stick deflection, WASD and buttons merge on
    top of a real pad or stand in for a missing one, and F9 toggles capture —
    @LeandroSSB (#205)

Contributed: tests

  • The C fixtures run under MSVC: 25 tests that skipped on a
    Visual Studio-only machine now execute, and a missing compiler is a real
    skip instead of a silent pass —
    @vyanhursky (#179)
  • A standalone POSIX directory-search fixture: 300 abandoned searches,
    live searches running side by side, restart, and mixed closes. #190 carries
    the same fix —
    @vyanhursky (#201)

Maintenance

  • Linux builds again: the platform library links libm, xbox_memory_layout.c
    defines _GNU_SOURCE for dladdr, the SDL window answers #162's AA hook, and
    GNU ld links the umbrella's archives as a group (the APU and kernel call each
    other).

Things that move. #189 moves contiguous allocations, so an address you
hard-coded from an earlier run, such as an explicit RECOMP_APU_DSP_ACK, may
be stale; the new auto default exists for exactly that. #189 also makes
XBOX_STACK_BASE and its neighbours runtime variables, so glue that used them
in a case label or a static initialiser won't compile. Neither generated code
nor the template does. #205 changes what an XGSCNT read returns, from host
time to the sequencer's sample count, for every title that reads it.

Held, with the reasons on each PR:

  • #186 (lifter and disasm, apexJCL): overlaps #182, #183 and #198 (and the
    held #...
Read more

v0.13.1 — "Cut Short"

Choose a tag to compare

@sp00nznet sp00nznet released this 07 Oct 21:49

Three fixes from one contributor, all about where a function ends. A CRT
memcpy ended at its own jump table, leaving the arms after it with no owner.
A run of alias entries in a gap each claimed the whole gap. A wrapped body's
rename carried every call with it, past the wrapper meant to intercept them.
Each one compiles, links and runs; the first two lift the wrong bytes, and the
third runs the right bytes without the code the project wrote to go around them.

  • Arms that follow their inline jump table are lifted inside the function:
    it is extended over its tables when every path stays in range, instead of
    dispatching to an address with no body —
    @NoRain211 (#173)
  • Gap aliases stay inside their gap: each ends after its own body, within
    its section's file-backed bytes, rather than at the next start measured before
    the other aliases existed. On DOA3 that is 1.58M → 1.28M lines of C —
    @NoRain211 (#174)
  • Calls reach a wrapped function's wrapper: direct calls, tail jumps and the
    dispatch table go to recomp_manual.c's sub_X, and the sub_X_gen body it
    calls is always emitted —
    @NoRain211 (#175)

Held: #176 (static callback recovery) accepts plain constants that land
mid-function in unlisted code as function entries; on Wreckless one of them is
data lifted as code. #162 (the pushbuffer executor, rebased) has a walk that
can run past PUT and an unbounded 32-bit vertex index. Both are back with
their authors.

Checked on Wreckless (DSTEAL_JP): regenerated with 0 failures, the only change
to the function set is four jump-table arms now lifted as labels, and the boot
reaches the same point as on v0.13.0.

A clean run is now 641 passed / 0 skipped, up from 630, plus 5,843
conformance vectors with no mismatches.

v0.13.0 — "Given Back"

Choose a tag to compare

@sp00nznet sp00nznet released this 07 Oct 07:22

Burnout 3: Takedown is playable end to end, and eleven contributed PRs from
three new contributors land beside it. The batch keeps finding things the runtime took and never
returned. A heap that handed a whole freed block to the next small request
and passed guest pointers to the host's
VirtualFree, so nothing came back.
A directory search abandoned mid-way that kept its host slot after the handle
closed. A failed indirect call that rewound over its arguments before the
caller popped them again.
pushad/popad emitted as TODOs, so the
registers a routine saved were never restored. None of them fails where it
happens; each surfaces later, somewhere else.

Contributed: lifter and recompiler

  • pushad/popad were TODOs, so a routine that saved every register
    with them returned with its caller's registers clobbered —
    @vyanhursky (#167)
  • A failed indirect call to a caller-cleans function popped its arguments
    twice
    : the failure path rewound over them, then the caller's
    add esp, N did too —
    @vyanhursky (#169)
  • rep movs touching the hardware aperture used host memcpy, whose
    vector loads skip the element-sized accesses trapped device memory needs —
    @vyanhursky (#170)
  • Two kinds of join read a flag nothing assigns: swapped-operand float
    compares, and a result snapshot meeting a compare snapshot. Each edge now
    computes the join's condition itself —
    @BearddOddity (#159)
  • Manual entry hooks: a sub_XXXXXXXX_enter() in the manual file runs at
    the top of the generated function on every call, direct ones included —
    @vyanhursky (#168)

Contributed: kernel

  • Opt-in memory reclaim and high reservations. RECOMP_HEAP_RECLAIM splits
    freed blocks, makes NtFreeVirtualMemory and MmFreeContiguousMemory
    actually free, and zeroes decommitted pages; RECOMP_EXT_VMA honours a
    reservation at a fixed address above the RAM mirrors. The heap table also
    gains a lock —
    @BearddOddity (#158)
  • Opt-in guest synchronisation. RECOMP_TITLE_KEVENTS gives a KEVENT a
    title built by writing its header a host event, instead of every wait on it
    failing at once; RECOMP_GUEST_LOCK runs one guest thread at a time —
    @BearddOddity (#160)
  • Closing a directory mid-search leaked the host search, and a reused
    handle value continued the old cursor —
    @vyanhursky (#171)
  • Kernel small fixes: NT pseudo-handles sign-extended on x64 (the CRT had
    retried NtDuplicateObject forever), STATUS_CONFLICTING_ADDRESSES → 487 so
    the CRT heap grows elsewhere, a 64-bit kernel call counter that no longer
    starts logging every call after 2^31, an OHCI short-packet underrun, MinGW
    thread-locals, two headless Ghidra scripts and a docs index by symptom —
    @BearddOddity (#157)
  • Diagnostics: g_xbox_path_hook, an [EXIT] line when a title ends
    itself, and an opt-in RECOMP_CALL_PROFILE —
    @BearddOddity (#161)

Contributed: build

Toolkit fixes found on MechAssault, Wreckless and Half-Life 2

  • A switch arm's resync left misaligned decodes inside it: MechAssault's
    CRT memcpy ran an inc esp lifted from the middle of a real mov, and
    every 28–31 byte copy returned with ESP a byte off — @sp00nznet (#135)
  • Jump tables named by their last slot were measured as one entry, so
    memmove's backward tail lost every arm after the table, epilogue included — @sp00nznet (#134)
  • The save directory was relative, so SHCreateDirectoryExW created
    nothing and a title probing partition1 at boot quit to the dashboard — @sp00nznet (#133)
  • String-reference labels no longer name code (#163); disasm trusts
    only seeds a run actually reached
    (#164); clang-cl builds the NV2A shim
    (#165) — @sp00nznet

Burnout 3: Takedown, from its entry point to every game mode playable

One and two players. Most of what it needed is general: the XDK's USB and
DirectSound stacks handing the hardware physical addresses, an x87 that
honours precision control, and enough of the NV2A to draw a 3D game through
its own D3D8LTCG. All by @sp00nznet.

Lifter

  • x87 results ignored precision control. Under PC=24 (D3D's default) the
    significand now rounds to 24 bits (#149)
  • lahf lifted; unordered x87/SSE compares set ZF, PF and CF, not
    "greater" (#150)
  • All eight SSE compare predicates, 4-lane sqrtps/rsqrtps/rcpps,
    and bare cmps/scas (#151)
  • Switch tables whose displacement is not slot 0: the CRT memcpy's
    unaligned-byte dispatch no longer silently skips the copy (#140)

Kernel, memory and interrupts

  • The RAM mirror span no longer leaks 1.8 GB of reservation on Windows (#138)
  • Physical page 0 is kept for the XDK USB stack (#143)
  • ISRs and DPCs run at their IRQL and restore it; KPCR.Irql is published
    (#148)
  • The DPC queue is locked, and KDPC.Inserted is honoured (#155)

Audio

  • GP/EP DSP memory is left untrapped under RECOMP_AC97_READY (#139)
  • The DSP doorbell ack runs whatever the front end is doing (#141)
  • The APU resolves physical addresses into the contiguous window and
    delivers its interrupt (#147)

USB

  • OHCI resolves physical descriptor and buffer addresses (#142)
  • RECOMP_PAD_SCRIPT and RECOMP_PAD_LIVE drive a pad headless (#144)
  • Up to four pads (RECOMP_USB_PADS), stick steps in pad scripts, and a
    rumble transfer no longer ends a control request (#154)

NV2A

  • Vertex programs, register combiners, four texture stages, visibility
    reports, near-plane clipping, bilinear filtering, and a threaded
    rasteriser (#152)
  • P8 textures sample through the stage palette (#145)
  • RECOMP_PB_REPORT_MS (#146); RECOMP_FB_DUMP_FLIPS dumps every flip for
    recordings (#156)
  • The window title shows the XBE title, FPS and draws (#153)

Held: #162, the pushbuffer executor from the X-Men Legends split, is a
draft and overlaps the executor that landed in #152; #128 is closed as split.

Merge fixups: #134's lifter half reduced to the one shape main's own fix
missed; #160's KePulseEvent block moved out of KeReleaseSemaphore, where
it never ran; #170's fixture renamed off glibc's index; #172's declarations
hoisted out of the macOS block so every POSIX host gets them.

A clean run is now 630 passed / 0 skipped, up from 579, plus 5,843
conformance vectors with no mismatches.

v0.12.0 — "Never Taken"

Choose a tag to compare

@sp00nznet sp00nznet released this 27 Sep 08:03

Forty-one contributed PRs, and the bug that keeps turning up is a branch that
compiles and can never go the way the guest meant. A
loop whose back edge
was always false, so every counted loop ran once. A
js after a byte
subtract that read a zero-extended copy and was never taken. A join reached by

test X, X on one edge and cmp X, 0 on the other that inherited no flags
and compiled dead. An
rcl/rcr pair lifted as a comment, which took every
64-bit divide with it. None of these fail loudly; each one is a path the guest
takes on the hardware and the generated C never does. Alongside them, a set of
bring-up instruments whose whole purpose is to make a silent path say something.

Lifter and recompiler correctness

  • loop/loope/loopne never decremented ECX and branched on a flag
    variable nothing assigns, so a counted loop ran its body once and wiped the
    tracked comparison for the jcc after it —
    @NoRain211 (#110)
  • After an 8- or 16-bit arithmetic or logic op, js was never taken and
    jns always was
    — the sign test read the zero-extended result at 32 bits —
    @NoRain211 (#120)
  • test X, X and cmp X, 0 could not meet at a join. They leave every flag
    identical, but the snapshot reconstructs them differently, so a block reached
    by one on each edge inherited no state and its branch compiled as never
    taken. SMT: Nine's video decoder has exactly that shape, and the dead branch
    smeared every picture horizontally —
    @fearkov (#122)
  • rcl and rcr were emitted as comments. MSVC's 64-bit divide helper uses
    them, so every long long division came out wrong —
    @fearkov (#104)
  • repe cmps/scas never set CF, so memcmp and std::string::compare
    never answered "less", and a zero-count compare reported "unequal" —
    @andeecollard (#124)
  • frndint rounded in the host's mode, not the guest's. The CRT's floor
    and ceil load a rounding control and call it, so floor(2.7) was 3 — JSRF's
    animation tables read one past the end —
    @andeecollard (#126)
  • An untranslated instruction vanished into a comment. It now carries
    RECOMP_UNIMPL(text, va), which logs the first time the site is reached and
    stops there under RECOMP_UNIMPL_TRAP=1. Existing game projects: copy
    recomp_unimpl() from templates/new-game/src/recomp_manual.c into yours,
    or the next regeneration will not link
    —
    @andeecollard (#117)
  • Switch arms past a function's recorded end had no generated body, so
    taking that case stopped on an unresolved indirect call (DOAXBV leaving View
    Collection). They get their own entries; existing bodies are byte-identical —
    @NoRain211 (#131)
  • A function reached only by address that opens with a switch was never
    detected
    , and its jump table was decoded as bogus functions —
    @andeecollard (#129)
  • Opt-in function coalescence, --coalesce-functions: merge fragments that
    function identification wrongly split, from explicit JSON bounds, refusing
    any merge it cannot prove. The series also hardens CFG recovery for every
    title — callback tables reachable only through recovered code, non-dword
    jump-table scales, iret as a terminator —
    @NoRain211 (#111–#116)
  • Indirect calls name their site, and a small, fully translated target set
    recorded at runtime becomes guarded direct calls on the next generation. A
    miss falls back to normal dispatch, so a guard can only be slow, never wrong —
    @andeecollard (#130)
  • Every MMX helper checked against the instruction it stands for, 4.7
    million cases — @fearkov (#105)

Runtime: kernel, USB, video, audio

  • Guest buffers are bounds-checked before the host touches them. A file
    read with a bad buffer or length walked the host past the end of guest memory
    and crashed inside the kernel; it now fails with STATUS_ACCESS_VIOLATION —
    @andeecollard (#89)
  • ADPCM blocks with a non-zero reserved byte were rejected, which hardware
    accepts — about 4% of JSRF's audio went silent —
    @andeecollard (#121)
  • USB: four root-hub ports, a 4 ms frame tick and the done-queue
    handshake.
    Ports 3 and 4 reset forever, enumeration raced the driver's
    timeouts, and the done queue was overwritten while the driver owned it. A
    raised IRQL now holds the interrupt off, bounded, so it cannot land in
    XAPI's half-built pipe setup; RECOMP_USB_HC picks the controller —
    @fearkov (#90, #91, #96)
  • The stuck-interrupt watchdog turned off a working pad after two seconds
    of ordinary 100 Hz reports; it now asks whether the driver acknowledged, not
    whether the status repeated. An unanswered GET_REPORT no longer ends input —
    @fearkov (#107, #118)
  • KeQuerySystemTime advanced in 15.6 ms steps, the host scheduler tick,
    not the console's resolution — @fearkov (#119)
  • NV2A: the flip methods were matched 0x18 too low, so the frame counter
    counted the wrong method and WAIT_FOR_IDLE was swallowed as a flip —
    @Heromachine (#101)
  • Every NV2A primitive number was one too low — strips drew as fans, fans
    as quads — @fearkov (#102)
  • Pushbuffer GET ran ahead of the executor, letting D3D overwrite commands
    not yet read, and a ring wrap dropped a submission per lap —
    @fearkov (#97)
  • YUV textures sampled as flat colour, alpha blending was ignored, and the
    window showed the frame being drawn instead of the one finished —
    @fearkov (#103, #108, #99)
  • Asynchronous reads answer STATUS_PENDING under RECOMP_ASYNC_IO, and
    a crash report with no stack left says so instead of printing an empty chain —
    @fearkov (#93, #95)

Bring-up instruments, all opt-in — @fearkov

  • RECOMP_WATCH=<va> names the guest code that writes a value (#92);
    --force-return ADDR=VALUE answers a function with a constant instead of
    hand-editing generated C (#100); RECOMP_DSP_ACK and RECOMP_POKE complete a
    DSP command word and hold a guest global (#98); RECOMP_PAD_PRESS and
    RECOMP_KEYBOARD stand in for a pad nobody has (#94, #106);
    RECOMP_TEX_DUMP_EVERY (#123); raised-IRQL holder tracing (#125); and the
    docs now recommend --split 250, since 1000 does not build in 15 GB (#109).

Also in this release

  • A "Runs, Never Draws a Frame" section in docs/pipeline/06-debugging.md,
    prompted by a Discord question about Black: the Wreckless lesson that a failed
    audio probe can skip a title's entire engine init, and which switches tell
    the cases apart.
  • Merge fixups: a per-thread floor in the buffer check that refused worker
    threads' reads; a fused test+jcc that disagreed with its own snapshot; an
    iret in data that aborted all of Wreckless's generation; MSVC builds of
    #125 and apu_mmio_hook.c.

Held: #128, the X-Men Legends bring-up, is careful work that changes
threading, IRQL, heap and APU behaviour for every title at once; it has been
asked to split so each part can be checked on the titles that already run.

A clean run is now 579 passed / 0 skipped, up from 386, plus 5,741
conformance vectors with no mismatches.

v0.11.0 — "Nothing Said So"

Choose a tag to compare

@sp00nznet sp00nznet released this 20 Sep 05:03

Twenty-seven contributed PRs, and almost every one of them is a defect that had
no voice. A pushbuffer executor reporting
draws 0 for a title submitting
geometry every frame. Thirty of thirty-two audio mixbins computed correctly and
thrown away, with no counter anywhere to say so. A 2,500-line APU that nothing
in the tree could call. A watchdog that could not read the register a title was
hanging on. An
__SEH_prolog detector whose "not found" was indistinguishable
from a CRT that has none. The previous release was named for the negative
control, the test that fails on purpose so a passing one means something. This
one is named for what happens when nothing is watching at all.

Seven lifter defects, found by differential fuzzing against an independent
x86 core
— each paired with a negative control that feeds the harness the
pre-fix expression and requires it to fail —
@andeecollard (#69–#72, #75–#77)

  • Rotates ran at 32 bits whatever the operand was. Every narrow read in the
    lifter arrives zero-extended, so a byte rotate happened inside a 32-bit word:
    the bits that should wrap at bit 7 landed in bits 31..8, and the store threw
    them away. ror al, 2 on 0x01 gave 0x00 where x86 gives 0x40. The count is
    masked to five bits and only then reduced modulo the width, so rol al, 16
    is the identity and came back zero. Same defect class as the sar width bug
    fixed two releases ago — that one was found and the rotates beside it were
    not — @andeecollard (#69)
  • bts/btr/btc reported the bit they left, not the bit they found. All
    four bit-test instructions copy the tested bit into CF, but only bt leaves
    it alone. The carry condition was rebuilt at the consumer by reading the bit
    a second time — which for the other three reads back what the instruction
    had just written. jb after bts was always taken, after btr never, after
    btc exactly backwards. That is the test-and-set idiom, "did I claim this
    or was it already taken?"
    , reading its own answer, with no input for which
    guest code could observe the truth. MSVC emits it for lock acquisition and
    for the character-map loops behind strpbrk/strspn/strcspn —
    @andeecollard (#70)
  • The sign flag after a compare was computed with signed overflow. js
    came out as (int32_t)(_fas - _fbs) < 0, and that subtraction overflows for
    exactly the inputs the sign flag is being asked about. cmp 0x80000000, 1
    leaves 0x7FFFFFFF on the hardware so SF is 0; in C it is INT_MIN - 1, and
    from -O1 the compiler is entitled to fold a - b < 0 into a < b and
    answer 1. Both gcc and clang do, so the emitted program's meaning changed
    with the optimisation level. Subtracting unsigned at the operand's own width
    and taking the top bit is SF exactly, at every width, with no undefined case
    — @andeecollard (#71)
  • popfd was listed as an instruction that preserves EFLAGS, next to
    pushfd, which genuinely does. popfd replaces every flag, so cmp eax, ebx; popfd; je resolved the branch from the comparison the restore existed
    to discard. The file already knew: the neg/sbb peephole four hundred
    lines away carries an explicit != "popfd" guard that the main tracking loop
    never got — two statements about the same instruction, disagreeing —
    @andeecollard (#72)
  • shld/shrd ignored x86's count rules, and a zero count wrote. The
    count is masked to five bits, and a masked count of zero must leave the
    destination alone. The emitted expression built src >> (32 - cnt), so a
    count of zero shifted by the operand's full width — undefined in C, and on a
    host that reduces the shift amount modulo the width it returns src whole,
    landing dst | src for an instruction that must not write at all.
    _lift_shift states the rule for shl/shr/sar next door; the
    double-precision pair never got it. Not latent: the title it was found on
    runs shrd eax, edx, cl twice with a runtime count —
    @andeecollard (#75)
  • movsd is two instructions and the dispatcher picked by name. The string
    MOVSD copies a dword from [esi] to es:[edi]; the SSE2 MOVSD moves a
    scalar double in or out of an xmm register. They share a mnemonic and nothing
    else, and the string branch runs first, so movsd xmm0, qword ptr [eax]
    walked esi and edi and touched neither operand the instruction names. The
    load never happened and the register kept its old value, silently —
    @andeecollard (#76)
  • The result-setter family rebuilt its condition at the consumer.
    and/or/xor, add/sub, adc/sbb, neg and the shifts all write
    their destination, and the jcc reading their flags can be several blocks
    later, so and eax, 0x0F; mov eax, 0x99; jne asked about 0x99. inc/dec
    already published their result into _fa at the write for exactly this
    reason, with a comment saying why — two instructions out of fourteen. This
    extends that rule to the rest —
    @andeecollard (#77)

Ten fixes from bringing up a real title, each one a place where the runtime
stopped a step short of something the title needed and said nothing about it —
@fearkov (#73, #74, #80–#87)

  • A loop head lost its only exit test. Blocks are lifted in address order,
    so the predecessor on a back edge sits after the block it reaches and has
    no out-state on a single pass. The join correctly refuses to guess, the jcc
    at the top falls back to _flags — a variable nothing ever assigns — and the
    branch compiles as never taken. Mid-function that costs a little accuracy; at
    the top of a counted loop it removes the exit. In the XMV decoder's row
    padding the loop stored eight bytes and advanced edi by sixteen forever,
    walked off the framebuffer and took the process with it. The state is settled
    to a fixed point before emitting now, and sub eax, ecx and dec eax are
    allowed to merge on the one thing a jz is actually asking —
    @fearkov (#86)
  • __SEH_prolog detection required the four-push form. The second byte
    marker is lea ebp, [esp+0x10], and that offset is not a constant — it
    counts the slots the helper pushed before it. The three-push form lands on
    0x0C and was undetectable. Silent, because "not found" is indistinguishable
    from a CRT that has no __SEH_prolog, which is a normal result: every SEH
    function then kept its caller's stale ebp, so the first frame-relative
    store landed in the caller's frame and the epilogue cut the stack back to
    it. A title can also link both forms, and returning the first match meant the
    winner was decided by nothing but the lower address —
    @fearkov (#87)
  • NV097_DRAW_ARRAYS was not handled by the pushbuffer executor.
    BEGIN_END arrives, END arrives, and in between comes a run description —
    first vertex and count — rather than the index list the draw path wants, so
    every batch was dropped with idx_count == 0 and the report said draws 0.
    A title submitting geometry every frame looked exactly like one submitting
    none. Decoding the run into indices turns the same seconds of the same title
    into 30,541 draws, checked against the pixel count rather than asserted —
    @fearkov (#81)
  • The emulated APU was unreachable. src/apu/ is a working ~2,500-line
    extraction of xemu's MCPX APU, and three independent gaps — each sufficient
    alone — meant nothing in the tree could call any of it.
    apu_hook_handle_mmio sits under a comment reading "called from VEH in
    main.c"
    and no main.c called it; g_apu_state was never assigned; and
    xbox_apu never linked xaudio2_8, which stayed invisible for as long as
    nothing referenced the archive — @fearkov
    (#84)
  • The AC'97 channel reset had to complete on the write. MSVC hoisted the
    load out of the wait loop, so the title reads the control register exactly
    once, a few instructions after writing it, and spins forever on that single
    value. A thread clearing the bit afterwards is racing a window a few
    instructions wide and gets one attempt; measured, it loses. Trapping the
    write — read-only page, single-step, mask RR out, re-protect — is the only
    version that is there in time — @fearkov
    (#82)
  • USB enumeration stopped one step short, six ways. A driver starting a
    fresh reset writes SetPortReset and ClearPortResetStatusChange in the
    same word, and the write-1-to-clear line ran after the handler set PRSC and
    wiped the bit that same write had just raised — so the port reset forever
    while looking connected, enabled and powered the whole time. Plus:
    descriptors live in the contiguous window the bounds check rejected, no frame
    clock, a control data stage that restarted every descriptor, only the control
    list walked, and a done queue never retired —
    @fearkov (#85)
  • RtlNtStatusToDosError answered 317 for every status it did not know.
    317 is ERROR_MR_MID_NOT_FOUND, "there is no message text for this
    number"
    — an honest default for an unmapped failure and the wrong answer
    entirely for a status that is not one. A resource loader that starts an
    asynchronous read and marks the object as loading only on ERROR_IO_PENDING
    never marked it, so the poll that finishes the load reported "not started" on
    every frame and the title sat in its first boot state forever with input,
    audio and rendering all working — @fearkov
    (#80)
  • **The watchdog could not ...
Read more

v0.10.0 — "Negative Control"

Choose a tag to compare

@sp00nznet sp00nznet released this 16 Sep 00:30

Twenty contributed PRs, and the thread running through them is the gap between
a thing being declared and a thing being true. A function declared with nine
arguments that takes ten. An argument declared on the stack that arrives in a
register. A shift declared arithmetic that was logical. A test suite declared
passing that was skipping. Several of these arrived with a
negative control —
a second test that feeds the harness the pre-fix code and requires it to fail,
on the grounds that a sweep passing against both spellings is testing nothing.
That is the right instinct, and it names the release.

Fourteen fixes from one contributor, found by driving a real title through
the pipeline
and chasing each wrong answer back to its cause — the largest
single batch the project has taken. Each arrived with a regression that fails
without it — @GTTeancum (#41–#45, #47–#51,
#53–#56)

Four kernel calls had the wrong ABI.

  • NtQueryDirectoryFile was declared with nine arguments and has ten. The
    missing FileInformationClass meant every argument after it was read one slot
    early — the search mask and restart flag came from the wrong places — and the
    bridge popped 36 bytes where the guest pushed 40. A four-byte stack leak per
    call, which is the kind that runs for a while and then does not. The
    enumeration also returned the host's . and .., which FATX does not have —
    @GTTeancum (#51)
  • KfRaiseIrql and KfLowerIrql are __fastcall, so their argument
    arrives in CL and the bridge was reading it off the stack. The cleanup table
    already said zero bytes, so the value read was whatever sat at that address —
    @GTTeancum (#54)
  • Counted object names were read as NUL-terminated. The XDK passes a
    directory name with a trailing wildcard and Length shortened to exclude it,
    without writing a NUL at the new end, so the path picked up the wildcard and
    whatever followed it in guest memory. The same fix made
    ObjectAttributes->RootDirectory real rather than always NULL, which is what
    lets a title open a save file relative to the directory handle it just
    enumerated — @GTTeancum (#53)
  • MmGetPhysicalAddress returned the virtual address unchanged. For the
    contiguous arena — the virtual window onto physical RAM — that hands a DMA
    consumer an address with the high bit still set —
    @GTTeancum (#55)

Three pieces of kernel state that were the wrong shape.

  • The pending kernel-dispatch slot was a single process-wide global. Two
    threads resolving an import at the same time raced, and the loser invoked the
    other thread's service, popping that one's argument count off its own stack.
    Thread-local now: a one-word change, and worth finding —
    @GTTeancum (#43)
  • IdexChannelObject was exported as an opaque self-pointer. It is a
    structure, and guest file-close code walks DeviceQueue.DeviceListHead at
    +0x28, where it found nulls. Host-backed synchronous I/O never enqueues guest
    IRPs, so the honest answer is a correctly formed empty circular list —
    @GTTeancum (#45)
  • The EEPROM advertised mono audio with AC3. XC_AUDIO was set to
    0x00010001, and in that field 1 means mono. A title asked the console what
    it was plugged into and was told one channel plus an encoded output path that
    does not exist — @GTTeancum (#56)

Six x86 semantics the lifter had subtly wrong.

  • SAR shifted at 32 bits regardless of operand width. Every narrow read
    arrives zero-extended, so (int32_t) on an 8- or 16-bit operand never saw a
    sign bit and the shift was a logical one wearing an arithmetic cast. sar al, 1 on 0x80 gave 0x40 where x86 gives 0xC0 — a negative number halved into
    a positive one, which is how a fixed-point divide or a signed average goes
    wrong without ever faulting. The count is masked to five bits as x86 does, and
    CF comes from the sign-extended value so a count at or past the operand width
    still reports the sign bit. Found independently by two people —
    @GTTeancum (#50) and
    @andeecollard (#57)
  • INC/DEC destroyed the carry flag they are defined to preserve. The
    whole point of inc over add reg, 1 is that CF survives it. Their own flags
    were no better: js, jl and friends re-read the destination at the branch,
    so a mov in between changed the answer, and OF and PF were not modelled at
    all — @GTTeancum (#44)
  • jbe and ja after and/or/xor were folded to constants. Those
    instructions do clear CF, so jb and jae after one really are 0 and 1 — but
    jbe is CF|ZF and ja is !CF && !ZF, and ZF is whatever the result was.
    Collapsing all four meant and eax, eax; jbe never branched and ja always
    did: wrong exactly when the result is zero and right the rest of the time, so
    it survives ordinary traffic and then takes the wrong arm on the empty list,
    the null handle, the zero count —
    @andeecollard (#57)
  • FIST/FISTP ignored the guest's rounding mode, lifting to llrint,
    which rounds by the host's. The era's CRT _ftol sets round-toward-zero and
    then converts, so the truncation it asks for silently became round-to-nearest:
    the 255.5 a colour-packing path expects to floor to 255 became 256, and every
    channel of white wrapped to 0 — @GTTeancum
    (#47)
  • FXAM was not implemented, and the status word was rebuilt from scratch at
    every read.
    fnstsw derived C0/C2/C3 from the comparison result alone, so a
    classification instruction contributed nothing and a title asking "is this a
    NaN, a zero, an infinity?" got the last compare back. The condition bits are
    shared x87 state now, so they survive a call the way the hardware's do —
    @GTTeancum (#49)
  • A classic push ebp; mov ebp, esp prologue pushed an uninitialised C
    local.
    The generated ebp was only seeded from the caller's frame for
    frameless functions, and a function with a real prologue reads it on its very
    first statement, so the guest's saved-frame chain got an indeterminate word
    that the epilogue pops back. Reached from both directions and merged together:
    one seeds it from the caller's frame, the other initialises the declaration,
    and the fix wants both — @GTTeancum (#48) and
    @andeecollard (#57)

Two recovery passes that were guessing. Comparison snapshots were discarded
at any control-flow join whose predecessors compared different registers — the
operands differ but the runtime slots they save into do not, so a shared consumer
can use whichever path actually ran (#41). And a bare immediate could split an
instruction the sweep had already decoded: an integer constant landing in an
unclaimed code gap is weak evidence, and treating it as a function entry carved
the real instruction stream in half (#42) —
@GTTeancum

Display gamma ramps. SetGammaRamp and GetGammaRamp were empty stubs that
took their arguments and dropped them, so a title that dims the screen for a fade
simply did not — invisible until you know the fade is missing rather than
instant. Implemented as a presentation-time transform, which is the part worth
having: the ramp is applied on the way to the swap chain and the guest's own
pixels are snapshotted first and copied back after, so a title that reads its
backbuffer back still sees what it drew, and a failed Present does not leave
gamma baked into guest memory. Runs on a deferred context so the game's bound
pipeline survives, and an identity ramp costs no GPU work at all —
@NoRain211 (#46)

The conformance suite no longer needs Windows to prove anything. It proves
the lifter correct by running each snippet as real x86 and comparing, which meant
a 32-bit MSVC — everywhere else it skipped, and a skip proves nothing. A
linux/386 container now supplies the toolchain and the CPU while the lifting
stays on the host. Two details make it trustworthy rather than merely green:
what is substituted is the toolchain and never the comparison, and the corpus and
XBE phases, which genuinely need PE linking, report as skipped rather than
passed. The sharpest find is in the harness — the native side deliberately runs
the x87 at 53-bit precision, and musl's i386 libm needs extended precision for
its argument reduction, so leaving that in force made cos(100.0) come back as
-1.27e16 — @dplewis (#52)

The Darwin TODOs became implementations. IsDebuggerPresent reads
P_TRACED on macOS and TracerPid on Linux instead of answering "no"
everywhere; SecureZeroMemory uses memset_s; GlobalMemoryStatusEx assembles
its answer from hw.memsize, the Mach VM statistics and vm.swapusage. The best
of them is anon_map_fd, which on macOS had been returning a literal 0 — a
valid file descriptor, and specifically stdin, so every file mapping on that
path was quietly backed by the wrong thing. Plus
InterlockedCompareExchange64 and waitable-timer handles —
@dplewis (#38, #39, #40)

Also. Four defects were fixed on integration, all on paths the Windows build
cannot reach — so none were wrong at review time, they were unbuilt. One PR
called __debugbreak(), an MSVC intrinsic, from inside the half of
`win32...

Read more

v0.9.0 — "Quietly Wrong"

Choose a tag to compare

@sp00nznet sp00nznet released this 12 Sep 19:20

A release of contributed fixes, and nearly all of them share a shape: the code
ran, returned, and was wrong, with no error anywhere. A stub that answers 0. A
flag that was dropped instead of preserved. A value rounded the wrong way. A
blend state that failed to create and left the previous one bound. None of them
look like a bug from where you find them.

Every kernel ordinal is routed. All 371 Xbox kernel exports — 347 function
ordinals plus 24 data exports — now have either a real bridge, a documented
stub, or a data entry. The ~136 that were unrouted fell through to a silent
return-0, which is worse than a crash: the title carries on with a plausible
answer it never asked for. The structural piece is a guest-VA to host-HANDLE
shadow table — a KEVENT/KSEMAPHORE/KMUTANT created through
KeInitializeEvent lives in guest memory and is not a handle, and
KeSetEvent and the KeWaitFor* pair had been treating the VA as one. The
audit that was supposed to catch all this was itself broken and passing: its
regexes anchored on a function's name, the file gained a comment mentioning
that name, the comment matched first, and the check was skipped entirely —
@DarthSidious666 (#32)

Two generator bugs that stop the build. cmovcc reads CF exactly as a
jcc does, but the carry-declaration scan looked only at jcc and setcc, so
a cmovb after an add emitted if (_cf) with _cf never declared. And a
guest function whose recovered name is a reserved C identifier or a Win32
export collides at compile or link time — Black has a function literally named
onexit (C2373 against UCRT's), Nightfire re-exports shims named exactly like
the APIs they wrap (LNK2005 against kernel32.lib). Both take the _<addr>
suffix func_id already gives duplicate names —
@DarthSidious666 (#28)

MMX was losing comparisons and rounding by hand.

  • Fifteen implemented MMX forms were missing from the EFLAGS-preserve set,
    so the lifter dropped the live comparison before them and recomputed. cmp eax, 0; pavgb mm0, mm1; sete al returns 1 on the CPU and returned 0 lifted —
    @NoRain211 (#34)
  • PADDUSW/PSUBUSW became TODO comments while the MOVQ loads and
    stores around them still ran, so a store published the unchanged value —
    @NoRain211 (#33)
  • Float-to-MMX conversion added 0.5 and cast, rounding halfway away from
    zero regardless of MXCSR, and range-checked against a float INT32_MAX
    that rounds up to 2147483648 and admits an out-of-range cast. Uses the SSE
    scalar conversions on x86 now — @NoRain211
    (#35)

Two D3D8 states that were wrong in the invisible direction.

  • Colour blend factors were copied into the alpha fields, which D3D11
    rejects, so a guest SRCCOLOR or DESTCOLOR failed CreateBlendState with
    E_INVALIDARG and left the previous state bound — a wrong blend rather
    than a missing one — @NoRain211 (#36)
  • D3DFVF_XYZRHW threw RHW away and emitted clip W = 1, so pretransformed
    geometry landed in the right place with its texture coordinates interpolated
    affinely across it — @NoRain211 (#37)

The POSIX build works again. Missing includes that C99 turned from warnings
into errors, strtok_s where POSIX wants strtok_r, and no implementation at
all for GetFileSizeEx or the Slim reader/writer locks. An SRWLOCK is usable
straight from SRWLOCK_INIT and is by definition taken from several threads
with nothing else held, so unlike the condition variables its first use
genuinely races, and it is serialised accordingly. Also caught the FATX
geometry constants being defined inside the _WIN32 half and referenced from
the POSIX half — @dplewis (#27)

A real flip drives the frame counter. FLIP_STALL now advances every
registered swap counter, and while those arrive the 62 Hz fallback timer stands
down. That timer exists for a title nothing presents for; once the pushbuffer
executor is actually running flips it is the wrong clock and an actively
harmful one. Half-Life 2's loader paces its intro video on this count, so a
62 Hz timer against an executor managing a fraction of a frame per second ran
the video forward in virtual time far faster than it could be drawn — which
looks exactly like a stalling, blocky video rather than a clock running away.
The rasteriser's per-pixel surface check moved to once per batch alongside it:
dma_resolve walked the arena high-water mark twice per pixel to guard a
rasterisation cheaper than the guard, and neither answer can change mid-batch.

An IDA path for name recovery, alongside the Ghidra one. Not a port of
pcrecomp's four IDA scripts — one exporter that writes the same
functions.json/symbols.json merge_names.py already reads, so the merge,
the placeholder filter, the sanitising and --apply stay where they are.
IDA's FLIRT and Ghidra's FidDb are the same idea with different coverage and
neither is a superset, so running both and taking the union names more than
either alone. merge_names learned IDA's autonames while it was there — loc_
is IDA's LAB_, and jpt_/algn_/asc_/stru_ have no Ghidra equivalent,
so without them an IDA export merges thousands of addresses-in-disguise into
the recompiler.

Also. write_if_changed in the translator — a regen rewrites all 54 chunks
of generated C, and an mtime bump on identical bytes costs a full /O2 rebuild
of 365 MB for nothing.