fix: defer V86-mode VBE disable until a legacy mode-set is observed

[felixrieseberg]

I don't know if you want this PR, it's very Windows 9x-specific. I'm opening it because I've enjoyed your code so much and I'll keep it in my fork but you might not want it - this is not strictly a bug fix, just a difference in behavior between real hardware and v86.

When a Win9x windowed DOS VM calls INT 10h, vgabios runs in V86 mode and its mode-set begins by writing dispi[4]=0. The Win9x VDD virtualises the standard VGA ports (3B0-3DF) for windowed VMs but passes 1CE/1CF straight through, so the VBE disable reaches the hardware while the rest of the mode-set (CRTC/seq/gfx/attribute writes) is captured into the VM's virtual register file. v86 then drops out of LFB rendering with the legacy registers still holding the SVGA values, and the screen shows planar garbage until the user manages to Alt+Enter back.

What this PR now does

Defer clearing svga_enabled when dispi[4] is cleared from V86 mode (EFLAGS.VM set); commit it only when an attribute_mode write actually reaches us. A real passthrough mode-set (fullscreen DOS, display-driver mode change, X.org int10) writes attribute_mode immediately afterwards, so the disable lands one register later. The windowed-VM leak never follows up, so the LFB stays live and the desktop keeps rendering with the DOS box in its window.

The enable path now also reapplies set_size_graphical whenever dispi[4] is written with bit 0 set, so a deferred-disable -> reconfigure -> enable sequence still resizes (set_size_graphical is a no-op when nothing changed). dispi_enable_value is updated either way, so guest read-back is unchanged.

Ring-0 / real-mode dispi writes (Linux bochs_drm, vesafb, bare DOS) have EFLAGS.VM clear and are unaffected.

[copy]

Do you have instructions for reproducing the problem? And is the bug being fixed present in qemu?

[SuperMaxusa]

Do you have instructions for reproducing the problem? And is the bug being fixed present in qemu?

Run an MS-DOS Prompt in Windows 9x with VBE9x (I used an old Windows 95 image from copy.sh/v86).

On QEMU and v86 (4e4bf55), Windows shows a screen with glitches:

Details

With this patch, it works properly. However, there is a bug when you exit from fullscreen mode, something similar to this: #1511.

IIRC, vmdisp9x fixes this problem with DOS VM by hooking up text-mode change and not doing it actually.

[felixrieseberg]

@supermaxuser is exactly right, I made the change to make it easier for people to run their old DOS games and apps without visual glitches.

I don't think v86 is wrong per se, I won't be hurt if you close the PR - I feel like I'm essentially working around a Windows oddity with graphical mode changes - but it's nice that you get working graphical/text switches without needing to patch Windows!

[copy]

It's a terrible hack (vga code looking at cpu mode), but it looks reasonably safe, passes all tests and fixes a real problem -> merged.

Thanks!