XMSDSK conflicts and crashes DOS4GW (DOOM) if set over 15168 size.

[flynnsbit]

Platform: Dos 7.1 (Win98SE DOS only)

Memory Config: HIMEM with HIRAM v1.9a (EXCLUDE=CE00-CFFF set)

Best game to test: Doom 1.9 with DOS4GW used as the extender (note using DOS32A works around the issue, but requires manual game changes which are not necessary on real hardware. Could this be a symptom of the problem we are seeing with DOS4GW compatibility on the core?)

Issue: Depending on the size of the XMSDSK you set, you can cause different issues with DOS4GW and doom (assume others but not tested) Examples are Doom locking up the core when exiting (anything over 17000), locking the core when starting (15170) , Causing a general protection fault with DOS4GW (15500), or scrambling the video screen with asci characters (16000)

Configurations: XMSDSK y: 15168 /y functions property any numbers referenced above will exhibit the errors or locking of the core based on the memory consumed by xmsdsk.

Images and video demo of the issue:
https://www.youtube.com/watch?v=XLJ8iE5u_zM

Example VHD for testing, use HIRAM and edit the autoexec.bat to remove the REMed entry for XMSDSK with the appropriate value.
IDE 0-0 BOOT-DOS98.zip

Shareware doom: doom19s.zip

[kblood]

Might this generally be why so many games works better with DOS32A ?

DOS32A is better than most DOS4GW. I read up on DOS4GW and most games have some cheap version of it. DOS32A is compiled in assembler I think it was, which is why it can be so much smaller than DOS4GW and that also makes it use less memory. But maybe more importantly, it contains a lot of fixes that have come up over the years. Overall handling memory and such better. So DOS32A is able to handle the problem, but DOS4GW expects the system to behave in a rather exact way.

Generally I think its probably a good idea to use DOS32A whenever possible, but it would be nice if AO486 would not require it as it does now.

[sorgelig]

If someone knows the root of issue then welcome to fix.
Btw, DOOM is the one of few games i'm using a lot for tests. Never had such problem. It may be related to EMS/XMS manager. I use QEMM.

[flynnsbit]

Right, The intent of this bug was not to call out what works (I also use QEMM for a lot of things). The intent was to show a very specific and re-producible bug that does not happen on real hardware when you use a combination of HIRAM, XMSDISK, and doom. The hope is it would give insights into why DOS4GW is so unstable on the core (50/50 shot in my testing). DOS32a more than 90% corrects the issue and allows for the game to play. There are 720+ games in DOS that require DOS4GW and if it worked exactly like it does on real hardware we wouldn't have to rely on DOS32a so heavily and only use it for edge cases.

Note, QEMM does not necessarily fix those game issues with DOS4gw and still requires a new stub using dos32a. This bug is not about doom or getting it to run. It is about a memory space overlap that seems to occur in certain scenarios. The goal here is to increase DOS4GW compatibility on the core as it is very far off from any other system (486 hardware or DOSBOX/BOCHs). You can test this by simply taking my packs and mounting them on real hardware or even in dosbox.

As sorg says, if a dev knows the root cause it would be great to look into that and I hope my reproducible steps outlined above give you some insights into the issue.

I am throwing out an educated guess here, but it could be related to having 256MB of memory in DOS and the memory manager or XMSDSK, DOS4GW isn't handling that well. One option that would give me more insights and something to work with is a 32MB memory option as 16MB isn't quite enough for everything we do (more memory to work with while still constrained in the bounds of dos). Is that an option? Notice how while set to 256MB in the core, we are able to negitively affect game loading that use extended memory by setting an XMSDSK in memory that is right around the 16MB window? "(anything over 17,000), locking the core when starting (15,170) , Causing a general protection fault with DOS4GW (15,500), or scrambling the video screen with asci characters (16,000)"

Is that a coincidence?

[sorgelig]

memory limit is based on BIOS Int 15h return. Regardless 16MB or 256MB option, a whole 256MB is available always. If you want to have other memory limits, then you can try to alter BIOS sources to report required amount.

[kblood]

I have tried looking into how to do FPGA development, but its very different from what I usually work with, which is .Net C# mostly. I am also just giving my input on it if it might help someone to figure out what is causing it.

Overall it might not be a critical problem since there usually are workarounds. I suspect every game that has these DOS4GW issues, its something that can be solved with one of the many memory managers and / or DOS extenders. DOS32A and HX DOS Extender combined probably solve most issues. And yeah, its not about Doom, its about what seems to be a AO486 handling memory differently... somehow. Might be the memory limit thing.

In my experience DOS32A seems to fix most of these problems... just hard to tell if its even the same problem when it does not fix it.

[flynnsbit]

Might this generally be why so many games works better with DOS32A ?

DOS32A is better than most DOS4GW. I read up on DOS4GW and most games have some cheap version of it. DOS32A is compiled in assembler I think it was, which is why it can be so much smaller than DOS4GW and that also makes it use less memory. But maybe more importantly, it contains a lot of fixes that have come up over the years. Overall handling memory and such better. So DOS32A is able to handle the problem, but DOS4GW expects the system to behave in a rather exact way.

Generally I think its probably a good idea to use DOS32A whenever possible, but it would be nice if AO486 would not require it as it does now.

Right, DOS32A does work very well, but I just spent 14 hours straight fixing games and creating new stubs with native DOS32a built into them, and that was on 300 games total (and only a mix of DOS4GW games). There are more than 7200 games "available" and no real way to find all of the DOS4GW as some games imbedded, some games included a DOS4GW.exe (720 games) some games don't and you would only know when you launch the game and get the copyright banner on start-up.

We could get through every game as a community, or create a script that auto stubs each game and then figure out how to share that and then figure out what games are now broken because of DOS32a. Or we could figure out what is causing so much instability with DOS4GW when compared to any other 486 emulator or PC there by increasing the likely hood that DOS4GW is going to work out of the box.

I am an expert on DOS32a at this point. It's just work, but it is a lot of work to find the right file, replace, test.

[kblood]

Yeah... that is a pretty big job. Hmmm... I wonder if there would be a way to scan EXE files for DOS4GW? I think in many cases the size of the EXE file helps reveal if it might use DOS4GW... but only in some cases. With the 720 games I think it would be safe to batch replace all of them with DOS32A.

But why build stubs into them? Wont it make more sense to run those games with DOS32A as a launcher? Like "DOS32A game.exe"? That works whether the DOS4GW file is external or internal. In this case it would be nice if there was an option to have it check for DOS4GW and then unload itself if none is found, but I am guessing that might not be simple to do.

But also with that solution you should be able to make the games launchable just with scripts. Some games are tricky though, and seems to require one file to run the game then have some launcher or something before getting to the DOS4GW file, and in those cases I am not sure if a script solution would be possible, unless that script would stub the file in need of it directly.

If just DOS32A could be used to run games even if they did not use DOS4GW. Then that could make for a quick fix for most of it I guess, but its not a very clean solution. Another way to automate would be if it was possible to scan exe files for whether they do have DOS4GW... hmmm... I did the DOS collection. I might now have time this week, but I could probably write a script that tried to use DOS32A on every EXE file it can find in the ExoDOS collection and then try to make a list of the ones that it worked on. I can probably not expect to be able to get any output from DOS32A or the game, so... I guess checking the size of the file I run it on would be the one way to do it. Then check the size again after.

At least this way we could map what games do use DOS4GW.... if I can get that to work. But with some games I suspect the DOS4GW might be hiding in something other than the exe file. Have you come across that?

But another way to rule out quite a few games, should be to just check the games size. DOS4GW should not be able to be below 250kb in size... so a game smaller than... I would guess even 500kb, would be highly unlikely to be based on DOS4GW. If the EXE file is larger than 640kb I would argue that increases the chance of it having DOS4GW embedded.

I might have misunderstood your process here. I did not try the thing you developed yet for the ExoDOS collection. Is it a bit like the ExoDOS converter that helps prepare it? So you are already trying to use scripts to stub what needs to be stubbed after installing the collection instead of needing your image of the full collection?

[kblood]

Oh... this part of DOS32A does seem capable of reporting the type of extender a certain EXE uses:
http://dos32a.narechk.net/manual/index.html
But I wonder if this SUNSYS Bind Utility can output something that I might be able to collect with some .NET based console utility. I think I could figure out something. That way I might be able to make a database over all the exe files and list what type of extender they use, or whether they do not seem to use one. I just have to figure out how to catch that output and interpret it with the program I could write.

[flynnsbit]

Finding them all would be huge for working around this issue documented here. Then it could be scripted as a post process before VHD creation.

As you said, some are not built into the main exe, and just call another game.dat, game.new, random file name. That is where creating the stub helps a lot. Sometimes there are three batch scripts deep before you get to the actual exe so you have to follow the rat hole of scripts before you get to the script where you should actually add dos32a to the front of it. That works, but is 3x the work from just replacing the stub in the EXE or creating a new one, and replacing whatever exe is there with it so all the scripts just reference it.

1_start.bat calls --> Run.bat and it calls --> Descent.bat --> and it calls D2.bat --> calls Descentr.exe --> which might not actually be the exe that needs to be stubbed and could call games.new or games.DAT. You could go down that rabbit hole, or you could just generate a new stub exe with sb /r game.dat. then replace descentr.exe with that and it all works up the chain (audio changes, screen resolution changes, etc that are built into the bat upstream). Thats why I build the real stub on MiSTer. (this is just an example and in this case descentr.exe actually has the stub, but there are other games that go this deep (C&C, etc)

[kblood]

I have begun unpacking ExoDOS v4. I have v5 downloaded, but I will use v4 to test with. Not sure if I can try it out today but I will report back when I figure something out.

[kblood]

I unpacked it all on a really fast SSD. But its probably best we move this to the forums. This is more of a workaround. DOS4GW often not working is an issue in AO486 either way. Maybe when we have mapped what games wont run we might be able to figure out exactly what is causing it. Either way, this should make the whole frontend creation easier to do.
Link the the thread I think it would be appropriate to debate this:
https://misterfpga.org/viewtopic.php?p=12897#p12897

I made a few posts in the thread. I think it worked. I created a CSV file based on my data mining. Here is the CSV put into Google Sheets:
https://docs.google.com/spreadsheets/d/17Vv37LfLiwk1vhCZGOIBjPv27wl_EvcIpd6Tatr4v1M/edit#gid=1324766038

I figured files that use DOS4GW seems to have the text "DOS/4G Copyright (C) Rational Systems" in them. Pretty glad I got a fast SSD, because I just did a search through all the files between 250kb and 100mb for that text.

[MacroFPGA]

Not sure if it helps, but from another forum it said

Certain versions of DOS4GW can only address 64MB

Anyways V2.00 of DOS4GW should have fixed the crash issue when more than 64MB is present. The latest version of DOS4GW is V2.01 and I think there was a thread around with downloads to all the available versions.

[flynnsbit]

Thanks MacroFPGA, a bit like we are doing now with Dos32a. This workaround could work for those games that don't work with 32a. Thanks.

[kblood]

I have been wondering if the 16MB option might fix some of these issues? I have not tested this idea yet myself, but I remember that it was probably introduced especially for Aladdin because it had a bit of a bug where it would not run at all if the machine had more than 16MB RAM.

[flynnsbit]

See Sorgs comments above on memory. We need a bios mod so that we can lock the memory to output a specific requested size. 16MB works for a few games (for other technical reasons) but using that option as the default breaks a lot of things when you are trying to cover a large set of games, launcher interface, and 4 types of memory managers. nevermind, found source.

[flynnsbit]

Best guess somewhere in either of these two spots, but I am not qualified to tinker here:

    case 0xe8:
        switch(regs.u.r8.al) {
         case 0x20: // coded by osmaker aka K.J.
            if (regs.u.r32.edx == 0x534D4150) {
                LOBYTE(extended_memory_size) = inb_cmos(0x34);
                HIBYTE(LOWORD(extended_memory_size)) = inb_cmos(0x35);
                extended_memory_size *= 64;
                if (extended_memory_size > 0x2fc000) {
                    extended_memory_size = 0x2fc000; // everything after this is reserved memory until we get to 0x100000000
                }
                extended_memory_size *= 1024;
                extended_memory_size += (16L * 1024 * 1024);

                if (extended_memory_size <= (16L * 1024 * 1024)) {
                    LOBYTE(extended_memory_size) = inb_cmos(0x30);
                    HIBYTE(LOWORD(extended_memory_size)) = inb_cmos(0x31);
                    extended_memory_size *= 1024;
                    extended_memory_size += (1L * 1024 * 1024);
                }

                LOBYTE(HIWORD(extra_lowbits_memory_size)) = inb_cmos(0x5b);
                HIBYTE(HIWORD(extra_lowbits_memory_size)) = inb_cmos(0x5c);
                LOWORD(extra_lowbits_memory_size) = 0;
                extra_highbits_memory_size = inb_cmos(0x5d);

                switch(regs.u.r16.bx)
                {
                    case 0:
                        set_e820_range(ES, regs.u.r16.di,
                                       0x0000000L, 0x0009f000L, 0, 0, E820_RAM);
                        regs.u.r32.ebx = 1;
                        break;
                    case 1:
                        set_e820_range(ES, regs.u.r16.di,
                                       0x0009f000L, 0x000d0000L, 0, 0, E820_RESERVED);
                        regs.u.r32.ebx = 2;
                        break;
                    case 2:
                        set_e820_range(ES, regs.u.r16.di,
                                       0x000f0000L, 0x00100000L, 0, 0, E820_RESERVED);
                        if (extended_memory_size <= 0x100000)
                            regs.u.r32.ebx = 0;
                        else
                            regs.u.r32.ebx = 3;
                        break;
                    case 3:

        case 0x01:
          // do we have any reason to fail here ?
          CLEAR_CF();

          // my real system sets ax and bx to 0
          // this is confirmed by Ralph Brown list
          // but syslinux v1.48 is known to behave
          // strangely if ax is set to 0
          // regs.u.r16.ax = 0;
          // regs.u.r16.bx = 0;

          // Get the amount of extended memory (above 1M)
          regs.u.r8.cl = inb_cmos(0x30);
          regs.u.r8.ch = inb_cmos(0x31);

          // limit to 15M
          if(regs.u.r16.cx > 0x3c00)
          {
            regs.u.r16.cx = 0x3c00;
          }

          // Get the amount of extended memory above 16M in 64k blocs
          regs.u.r8.dl = inb_cmos(0x34);
          regs.u.r8.dh = inb_cmos(0x35);

          // Set configured memory equal to extended memory
          regs.u.r16.ax = regs.u.r16.cx;
          regs.u.r16.bx = regs.u.r16.dx;
          break;
        default:  /* AH=0xE8?? but not implemented */
          goto int15_unimplemented;
       }
       break;
    int15_unimplemented:
       // fall into the default
    default:
      BX_INFO("*** int 15h function AX=%04x, BX=%04x not yet supported!\n",
        (unsigned) regs.u.r16.ax, (unsigned) regs.u.r16.bx);
      SET_CF();
      regs.u.r8.ah = UNSUPPORTED_FUNCTION;
      break;
    }

[wishusknight]

I wonder if someone would be willing to fork in another dos core and call it "AO386" with basically a small physical amount of ram like 8 or 16 mb. That way it can be its own thing. That is if this is the real issue.

[sorgelig]

16MB is already an option in OSD.

[flynnsbit]

I just wanted it stated here for all that by the very nature of the bug process being reported in the original post, that the 16MB option was not used for obvious reasons. We need more than 16, and the theory is less than 256 but there is no way to validate that without something in between.

[sorgelig]

16MB was a border at some time like it was for HDD with 500MB then 8GB.
Something between 16 and 256 has no real meaning as there were no borders between which should change any behavior. It's also visible from BIOS API where 16MB was a ceiling for max available memory returned by BIOS till new Int call was implemented.

[flynnsbit]

32

[flynnsbit]

16MB was a border at some time like it was for HDD with 500MB then 8GB.
Something between 16 and 256 has no real meaning as there were no borders between which should change any behavior. It's also visible from BIOS API where 16MB was a ceiling for max available memory returned by BIOS till new Int call was implemented.

I can just close this if you think this is not valid. Please let me know.

[sorgelig]

it's valid, but i don't have solution for now

[kblood]

Seems like a 64MB option would make sense, given it would then support all the games with the DOS4GW issue? I also think 64MB would be enough for anything but Windows software? Not that it should replace the 256MB option, but the issue reported here seems to be caused by the high amount of memory the core has compared to the actual computers from that time rather than something actually being wrong in the memory allocation and FPGA code.

[solskogen]

What about having 1MB, 2MB, 4MB, etc... as a option?

[wishusknight]

16 mb is the limit of ISA addressing. Dos 5 and 6.0 memory managers had a limit of 64 if i recall. 6.22 and himem that came with windows for workgroups handled more. Isn't the issue that the 16 mb option is not trully removing all ram but simply the bios reporting less? That would be an issue for applications with large ram issues that scan by address and try to write to area's that are unprotected.

[sorgelig]

Yes, it's simply BIOS reporting less

[sorgelig]

I believe there were workarounds for games having issues on larger RAM on PCs of that time. I doubt users of that time were physically removing the memory to play such games.

[wishusknight]

You would be correct. Most of the time back in the day we ran patchers or had secondary computers for the job. I am having my doubts its about what the bios is reporting. It could be just how it sees the hardware as being a contiguous block.

It wasn't really until late 1999 early 2000 that most of us had more than 64mb ram, and by then we were moving on in games with win 2000 and XP not even supporting them. It makes me now want to pull out one of my retro boxes from storage and test this out again. There was a 24 bit addressing limit back in the day for legacy machines, (16mb) and for compatibility there was always a seamless translation of 24 bit addresses to the 32 bit address space....

DOS/4G was painfully sensitive to dma handling in this aspect and was always a festering fusspot with certain motherboard chip sets even. And how it interacted with real mode and virtual mode on the 80386. Especially if a device on the ISA bus that needed DMA didn't play nice, like a cheap scsi card or a Gravis Ultrasound. Sometimes turning on a "Memory Hole" in the bios worked. And the bios was still allowed to report all ram, but usable ram would no longer be in one contiguous block. This would cause Dos 6 to only see 15 or 14 mb of ram depending on how it was set. And windows 9X could often overlook it....This didn't always work though. It really depended on the implementation of the DMA controller in the chipset and the MMU.

sorry for the nostalgic and pointless rambling. Im poking around trying to see if I am able to shed any light on this. FPGA implementation based on Bochs is quite different from the real thing from back in the day. DMA controllers such as the Intel south bridges from the TX chipset and onward were able to handle this translation far cleaner. But a "just enough" implementation is going to be a real adventure. Cyrix CPU's especially were also a huge nightmare. Even well into the MII days. They were essentially a clean room design based on principal operation and not all the workarounds Intel built into their hardware over the years. Much like Bochs and Dosbox today.

[sorgelig]

Btw, DMA in ao486 is classic ISA style, so only 16MB can be addressed by DMA.

[sorgelig]

Anyone is welcome to provide a correct RAM size properly detected in HW and BIOS. I will accept PR if it will really fix all these problems.

[luishg]

What about having 1MB, 2MB, 4MB, etc... as a option?

Just want to add that the memory cap is not a solution. You can try to limit memory to 16Mb through OSD and Warcraft II over its default DOS/4GW won't pass.

Btw Warcraft II specs:

Computer: Warcraft II requires a 100% IBM PC compatible computer, with a 486/33 MHz or better processor and at least 8 megabytes of memory. Operation System: You may play Warcraft II under MS-DOS version 5.0 or higher, Windows 95, or Windows 3.1.