This is a very exciting project and I thank you for having pushed it as far.
This fork is welcomed for DOS 5.00 on x64 WoW64 didn't appear a priority for Stephanos in the OpenNT forums.
This said lest not forget NTVDMX86 and similar prior x64 emulators such as DOSBox, NTVDM64, v86, vDOS and Boch cannot provide the precision (read instructions cycles compatibility) of a real virtual machine like VMware running original DOS.
perhaps the most important advantage of emulated DOS 5.00 emulator for i486 (no pentium+ opcodes like CMOV) lies within Insignia Solutions' SoftPC industrial robustness and relative isocompatibility Microsoft backed approval.
I have compiled successfully the said ntvdm following your guidelines -have yet to test it- but in brief : speed and accuracy (such as the BDA RTC 0:46ch) ontop of alluded robustness are expected to better most existing emulators.
Already the benefits NTVDMX86 bears for the demoscene sizecoding community is attractive enough though it's not the entire DOS part that is interesting but rather the long mode compatible BIOS/PC 16-bit emulator .com-spawning with thunk handling along its sources (?).
I have some remarks althoug again, I have not seen it working yet, nor did I use NT4 or v86 on NT4 extensively which also differs from your NTVDMX64 and I probably made errors below as my memory can be faulty (all errors are my own) :
The first caveat you mention is the obligation of having Windows download symbols for first run of a DOS program "because the loader code needs to fetch symbols from the Microsoft Symbol server so that it can call OS internal functions in order to properly start NTVDM".
This is not too enticing considering how notoriously Windows10 and all Windows x64 OS value user privacy, but is I suppose half-understandable if by symbols you mean DLL imports by ordinal vs. by name for Heaven's Gate startup/end injections in WoW64 hooking.
If this is indeed the case then maybe the README.txt wording seems confusing not making it clear enough this is just a one-time update at first launch and unrelated to any DOS program (which as any other programs if compiled with debug information might contain symbols too).
The second caveat you mention is that "currently there are crashes with various DPMI programs like i.e Borland Turbo C or some DOS extenders [...] maybe someone can help me debugging them?", and this is where I am a bit pessimistic about outcome.
Your very interesting blog hardwarefetish.com sports an "In-depth knowledge of the WIN32 Kernel and MS DOS" and you seem fortyish so you must be aware of all the hereunder :
-problem 1 : DPMI/Dos extenders already crashed relatively often on real DOS (thus not requiring "fixing" in this context)
Popular DOS-extenders include Phar Lap's, Borland's, EMM386 QEMM or more recent HX.
Dos extenders are not standardized in that not often can one be replaced by the other : more often than not a binary would only work with a specific dos-extender
All have their quirks : Phar Lap allows ring '0' GPF with -priv, Borland's requires you beforehand use their compatible linker, HX emulates Win32, others allow big/unreal mode specific to real 16-bit or make use of "standard" DPMI v1.0 API never implemented by DPMI v0.9 provider Microsoft.
-problem 2 : DPMI/Dos extenders crashing now and not on real DOS
Exempt MS distributed Loadhigh/LH variations handling as little as XMS, EMS or UMS memory remaining within first mega of memory (and SoftPC is expected to handle that minimally) and as far as I can recall,
Microsoft AND Windows (ie : DOS on Windows) only had two officialy distributed paths to DOS-related memory management : EMM386 and Enh386 world DPMI 0.9 services host acquired through Qualitas located mostly in IVT interrupt 31h.
Thus it sounds reasonable to only focus on these two alone and not on more exotic ones historically-wise and for the sake of same-context retro-compatibility.
EMM386 booted until Enh386 in which turn MS-DPMI took over to protected mode. EMM386 could be seen working in initial virtual 8086 mode afterwards in command.com up until Win98.
But the heart of any fully-featured DPMI/Dos extenders requires protected mode accessibility through keyboard A20 gate and in the case of NTVDMX86 SoftPC's emulated Ms-DOS version 5.00, initially destined for non-x86 architecturess, this A20 gate mechanism is unexpected.
Also turning it on in a non-virtualized environment (16-bit real emulation in a 32-bit protected mode) on amdx64 architecture means encountering same impossibility about 16/32 or 32/64 but not both at same time.
So to resume, as far as DPMI/Dos extenders go, only emm386 probably needs testing in this context (the rest either did not work correctly before or could not possibly correctly work in this 32-bit protected mode emulation).
Even when EMM386 semi-worked in V86 it was only in conjunction with MS' DPMI v0.9 flavour belonging exclusively to Enh386 world seperating real/protected 16-bit and protected 32-bit mode, unfound on any other SoftPC/NTVDM than x86.
In the case of x86 the SoftPC emulation passed on to v86 for execution - in the other architectures it emulated the execution but in return could not rely on Enh386 MS DPMI v0.9 and same for EMM386.
Obtaining NTVDMX64 by forcing emulation on x86 architecture probably stripped the Virtual-8086 "compatibility" of EMM386 with Enh386 DPMI v0.9.
The leftovers witnessed working for EMM386 or any other DPMI/Dos extenders are most likely linked to real mode memory handling and safely dealing with only upper, extended or expanded memories within RAM first megabyte.
If that is the case it would thus seem logical to envision only "patching" EMM386 and/or hooking SoftPC's IVT 31h to fail gracefully instead of crashing (or increase workability by providing fake 32-bit LDT selectors pointing to same constant SoftPC memory space).
I am still interested in seeing how to push the enveloppe but was unable to find any contact link (I can be contacted on ircnet in the evenings or through my blog -on which I plan to talk about your work).