Enable running in a VM

[BinaryMelodies]

This pull request enables running elksemu using the libx86emu software library. Additionally, it also contains the old vm86 syscall approach with code mostly lifted from the older version, with some adjustments to work with the newer code base.

What works:

• Three new compilation flags to switch between emulation modes: USE_VM86, USE_X86EMU, USE_PTRACE (see elksemu/Makefile for an explanation).
• Compilation and execution successful on an x86-64 Linux install.
• All 3 options have been tested, except the USE_VM86 binary could only be run under qemu-i386.

What does not work:

• vm86 under x86-64 Linux, there is no way to fix this, as the kernel does not support this syscall.
• ELKS signal handling seems to not work, even before this pull request was submitted. However this is likely due to the old dev86 toolchain I have been using. Making signal handlers near calls instead of far calls in the emulator has partially remedied the issue, but even that requires some bug fixes. (This change is not part of the current PR as it would likely break more modern binaries).

What has not been tested:

• New ELKS header support (far code, relocations), as I currently don't have access to such binaries. However this should still work, since the overall code structure has not been altered.
• USE_VM86 mode on an actual x86-32 Linux install.
• Older x86-32 Linux kernels with vm86old syscall only. The current pull request does not support them.
• Running it as a Linux a.out binary compiled with bcc. Makefile claims to support it, but I don't believe anyone has tried doing this since a.out support has been long dropped from the mainstream Linux kernel.

[ghaerr]

Hello @BinaryMelodies,

Wow! :) You've done a fantastic job both refactoring the old code and adding support for new ptrace on Linux-x64 and libx86emu. Thank you, this all looks great!

I look forward to trying this out on macOS with libx86emu as well. It's be great to (optionally) get rid of the Linux dependency on emulating ELKS binaries with elksemu.

After commit, I'm thinking of enabling USE_PTRACE=1 in Makefile, since elksemu is automatically built with an ELKS build as the last item built - this should work for all Linux-x64, right? It won't hurt if it doesn't build since everything else will have been built by that point. The master ELKS Makefile uses the following to build elksemu:

ifeq ($(shell uname), Linux)
    $(MAKE) -C elksemu PREFIX='$(TOPDIR)/cross' elksemu
endif

Not having a Linux box, I'm not sure what uname option might differentiate between 32- and 64-bit Linux systems.

Making signal handlers near calls instead of far calls in the emulator has partially remedied the issue

It's a bit of a long story how signal handling works in ELKS, but basically the C library registers a single far function with the kernel, and when a signal handler callback is required, the kernel fakes up the user stack to (temporarily) return to that C library function with the signal number on the stack. The library then uses the signal number to index the process' internal array of 16 possible near or far signal handlers (depending on the compiled application small/large model etc) and calls it (using the compiled application memory model default near/far). After the signal handler returns, the C library executes a final IRET which then restores CS/IP/FLAGS to the interrupted original code.

Let me know if you'd like any help or more information for getting signal handling to work in elksemu.

I currently don't have access to such binaries.

It's relatively easy to build ELKS and create all the application binaries, which all live under elkscmd/*/; use ./build.sh which will first build the ia16-elf-gcc compiler (once) which takes way too long, then the C library, kernel and applications are built in a few minutes. If you've got other things to do, I'd be happy to zip up all the /bin ELKS binaries for you instead.

New ELKS header support

Oh, I see - it seems that elksemu uses the Linux binfmt interpreter mechanism for a partial loader callback, then uses a special Linux-only search_binary_handler to actually load the a.out file. This definitely won't work on macOS - looks like I may have to write my own loader (starting from elks/fs/exec.c) in order to load extended header binaries, which I believe are ELKS-only and likely not supported by the Linux kernel. This would also all have to be rewritten to support loading OS/2 NE executables. I have some non-Linux a.out executable loading code in my 86sim project, perhaps I should dig that out and look at it further, if interested.

Running it as a Linux a.out binary compiled with bcc. Makefile claims to support it

I didn't even realize that was in there, and yes definitely that won't work! I suppose since I've not been able to use elksemu I've never gone in there to clean all that old mess out.

Thank you!

[BinaryMelodies]

Hello @BinaryMelodies,

I am not sure about the etiquette for answering a closed ticket, but I quickly wanted to respond to a few points.

Wow! :) You've done a fantastic job both refactoring the old code and adding support for new ptrace on Linux-x64 and libx86emu. Thank you, this all looks great!

I look forward to trying this out on macOS with libx86emu as well. It's be great to (optionally) get rid of the Linux dependency on emulating ELKS binaries with elksemu.

I am glad I could help out! And as long as the system calls translate well enough on macOS (or any other UNIX), it should work smoothly.

After commit, I'm thinking of enabling USE_PTRACE=1 in Makefile, since elksemu is automatically built with an ELKS build as the last item built - this should work for all Linux-x64, right? It won't hurt if it doesn't build since everything else will have been built by that point. The master ELKS Makefile uses the following to build elksemu:

ifeq ($(shell uname), Linux)
    $(MAKE) -C elksemu PREFIX='$(TOPDIR)/cross' elksemu
endif

Not having a Linux box, I'm not sure what uname option might differentiate between 32- and 64-bit Linux systems.

That's a good idea, I wasn't sure where to enable that. I don't see why it wouldn't work on any x86 based Linux, whether 32-bit or 64-bit.

I tested out uname, all of -m, -p and -i return x86_64 for me, and i686 in 32-bit mode.

Making signal handlers near calls instead of far calls in the emulator has partially remedied the issue

It's a bit of a long story how signal handling works in ELKS, but basically the C library registers a single far function with the kernel, and when a signal handler callback is required, the kernel fakes up the user stack to (temporarily) return to that C library function with the signal number on the stack. The library then uses the signal number to index the process' internal array of 16 possible near or far signal handlers (depending on the compiled application small/large model etc) and calls it (using the compiled application memory model default near/far). After the signal handler returns, the C library executes a final IRET which then restores CS/IP/FLAGS to the interrupted original code.

Thank you for the clarification! Maybe the lack of IRET is the issue here, but I would have to check what the old dev86 toolset generated, and I am not very well versed with signal handling. I am not sure if I will be able to resolve that yet.

Let me know if you'd like any help or more information for getting signal handling to work in elksemu.

I currently don't have access to such binaries.

It's relatively easy to build ELKS and create all the application binaries, which all live under elkscmd/*/; use ./build.sh which will first build the ia16-elf-gcc compiler (once) which takes way too long, then the C library, kernel and applications are built in a few minutes. If you've got other things to do, I'd be happy to zip up all the /bin ELKS binaries for you instead.

I had issues with compiling ia16-elf-gcc before and I had to use some prebuilt binaries. If you can send over the precompiled binaries, that would be very helpful.

New ELKS header support

Oh, I see - it seems that elksemu uses the Linux binfmt interpreter mechanism for a partial loader callback, then uses a special Linux-only search_binary_handler to actually load the a.out file. This definitely won't work on macOS - looks like I may have to write my own loader (starting from elks/fs/exec.c) in order to load extended header binaries, which I believe are ELKS-only and likely not supported by the Linux kernel. This would also all have to be rewritten to support loading OS/2 NE executables. I have some non-Linux a.out executable loading code in my 86sim project, perhaps I should dig that out and look at it further, if interested.

I actually haven't used, nor touched the kernel module at all, and I don't believe it's necessary for running ELKS executables. The elksemu executable was already writte to handle version 0 and version 1 headers, with relocation and far support, and I didn't change that at all. I just like to make sure the code still works after modification, but in this case I wasn't able to. I think the binfmt_elks.c is there only as another way to run ELKSE executables on Linux.

There is however still no OS/2 NE support. If you have some example binary lying around, I could use that to start working on NE support in elksemu (at least at some point down the line).

Running it as a Linux a.out binary compiled with bcc. Makefile claims to support it

I didn't even realize that was in there, and yes definitely that won't work! I suppose since I've not been able to use elksemu I've never gone in there to clean all that old mess out.

Thank you!

[ghaerr]

I am not sure about the etiquette for answering a closed ticket

No issues, we do it all the time!

I tested out uname, all of -m, -p and -i return x86_64 for me, and i686 in 32-bit mode.

Good information, I think uname -p might be useful for determining whether Linux (and macOS) is running as 64- or 32-bit. BTW, there's no uname -i on macOS.

Maybe the lack of IRET is the issue here, but I would have to check what the old dev86 toolset generated

No question that the signal library code in dev86 won't work with the current ELKS. Our current kernel changed the implementation from the older dev86 way because otherwise the kernel task structure had to contain far proc pointers for all 16 signal handlers, which used way too much kernel stack space. In any case, ELKS does not support using dev86 to create binaries, although almost all system calls have not changed, so some a.out's will run. (Thus, its entirely questionable whether ELKS should continue to support v0 binaries, as IIRC a system call or two number might have changed. But we'll leave that discussion for another day).

Thus, the only reasonable way to debug signal code would be to produce executables using the current C library. That library is now produced in three separate library/archive files: ia16 versions (actually one for each calling convention), C86 as86-compatible libc86.a, and OpenWatcom .obj compatible libc.lib. I can send you the latter two if you like. The ia16-elf-gcc libraries are built way deep in the gcc source tree build, and hard to pull out to make work standalone.

I think the binfmt_elks.c is there only as another way to run ELKSE executables on Linux.

Yes, you're right - I checked more thoroughly and the elksemu loader is in elks.c::load_elks(). The other function(s) are just there to run ELKS executables without running elksemu first. So it might not be that big of a deal adding support for OS/2 NE.

If you have some example binary lying around

Here's the C86 toolchain, most but not all are OS/2 NE executables, the rest are ia16-elf-gcc a.out extended headers. I just grabbed all of them:

Gregs-MacBook-Pro1:elks-bin greg$ file *
ar:       Linux-8086 executable, A_EXEC
as:       MS-DOS executable, NE for OS/2 1.x (EXE)
c86:      MS-DOS executable, NE for OS/2 1.x (EXE)
cpp:      MS-DOS executable, NE for OS/2 1.x (EXE)
disasm86: Linux-8086 executable, A_EXEC
ld:       MS-DOS executable, NE for OS/2 1.x (EXE)
make:     Linux-8086 executable, A_EXEC
objdump:  MS-DOS executable, NE for OS/2 1.x (EXE)

cc86.zip

I had issues with compiling ia16-elf-gcc before and I had to use some prebuilt binaries. If you can send over the precompiled binaries, that would be very helpful.

Here's the entirety of ELKS /bin, these are almost all ia16-elf-gcc a.out v1 binaries, with the exception of a few shell scripts.
bin.zip

I'd like to hear about any compilation issues on ia16-elf-gcc should you try again. I think we have a patch on an open issue regarding GCC 11, if that's what you're using.

Thank you!

[BinaryMelodies]

No question that the signal library code in dev86 won't work with the current ELKS. Our current kernel changed the implementation from the older dev86 way because otherwise the kernel task structure had to contain far proc pointers for all 16 signal handlers, which used way too much kernel stack space. In any case, ELKS does not support using dev86 to create binaries, although almost all system calls have not changed, so some a.out's will run. (Thus, its entirely questionable whether ELKS should continue to support v0 binaries, as IIRC a system call or two number might have changed. But we'll leave that discussion for another day).

Thus, the only reasonable way to debug signal code would be to produce executables using the current C library. That library is now produced in three separate library/archive files: ia16 versions (actually one for each calling convention), C86 as86-compatible libc86.a, and OpenWatcom .obj compatible libc.lib. I can send you the latter two if you like. The ia16-elf-gcc libraries are built way deep in the gcc source tree build, and hard to pull out to make work standalone.

Alright, if I get the time, I will look into compiling binaries using these tools. This could help updating the linker I am working on to support newer ELKS binaries.

I think the binfmt_elks.c is there only as another way to run ELKSE executables on Linux.

Yes, you're right - I checked more thoroughly and the elksemu loader is in elks.c::load_elks(). The other function(s) are just there to run ELKS executables without running elksemu first. So it might not be that big of a deal adding support for OS/2 NE.

If you have some example binary lying around

Here's the C86 toolchain, most but not all are OS/2 NE executables, the rest are ia16-elf-gcc a.out extended headers. I just grabbed all of them:

Gregs-MacBook-Pro1:elks-bin greg$ file *
ar:       Linux-8086 executable, A_EXEC
as:       MS-DOS executable, NE for OS/2 1.x (EXE)
c86:      MS-DOS executable, NE for OS/2 1.x (EXE)
cpp:      MS-DOS executable, NE for OS/2 1.x (EXE)
disasm86: Linux-8086 executable, A_EXEC
ld:       MS-DOS executable, NE for OS/2 1.x (EXE)
make:     Linux-8086 executable, A_EXEC
objdump:  MS-DOS executable, NE for OS/2 1.x (EXE)

cc86.zip

Thank you! This is fantastic, I will study these files to see if I can start implementing some OS/2 binary support.

I had issues with compiling ia16-elf-gcc before and I had to use some prebuilt binaries. If you can send over the precompiled binaries, that would be very helpful.

Here's the entirety of ELKS /bin, these are almost all ia16-elf-gcc a.out v1 binaries, with the exception of a few shell scripts. bin.zip

I am already noticing there seem to be some issues with some unimplemented system calls in ELKS, as well issues with brk. I will probably look into it a bit. But the executables seem to launch.

I'd like to hear about any compilation issues on ia16-elf-gcc should you try again. I think we have a patch on an open issue regarding GCC 11, if that's what you're using.

Thank you, I will let you know!