-
Notifications
You must be signed in to change notification settings - Fork 106
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[libc] Pass 2: Add system calls to OpenWatcom C library for ELKS #1912
Conversation
So we are heading towards a complete ELKS distro based on Whatcom compiler, but this time with more features? Build bigger programs? I remember the 64 kb limit was quite a restriction for porting even simple (in terms of functionally) programs such as editors. |
It's probably not worth trying to get all the existing ELKS applications recompiled and distributed with the Watcom compiler, as that would then require two major toolchain build and installs in order to build ELKS. Also, the kernel is being built with ia16-elf-gcc and has many associated ASM source (both internal to .c files as well as external .S) files that would need translating from 8086 AT&T syntax to Intel syntax. You gotta love two major incompatible formats for the x86 instruction set! Yes, the goal is to introduce the enhanced feature set of the Watcom compiler to ELKS application programmers. This would include large code and data models which should allow for much larger programs to be used on ELKS. The current work is just getting "regular"-sized (small programs compiled using large model) programs runnable on ELKS using Watcom. In order to support programs larger then 128K code or 64K data, the ELKS a.out executable format and kernel loader will have to be enhanced. I'm thinking about how that might be accomplished now. Eventually, the Watcom C compiler itself, along with the C library and linker tools, could possibly run directly on ELKS, which would be a major accomplishment, as then ELKS could conceivably become self-supporting with its own compiler, C library and kernel. |
Hello @ghaerr , Thank you for the great works. 2.1 You may use, reproduce, display, perform, modify and distribute Original |
At least we have already comment in our license comment, Software included beside the ELKS kernel (i.e. elkscmd) may be under different licensing terms (FreeDOS is also compiling with OpenWatcom so it would be ok just using it, but ...) |
Hi @ghaerr, I am curious whether the new compiler comes with a performance profiling tool. Or did I miss it, and we already had one with ia16-gcc? Best |
Hello @tyama501,
The OpenWatcom source
This section basically says that Personal Deployment is excepted from the further requirements of any commercial (or non-Personal) use/deployment of the software. Since ELKS users may or may not be using OpenWatcom and ELKS for Personal Use, I have followed the requirements for commercial deployment which is in Section 2.1:
This seems to be saying that commercial use of software built from OpenWatcom source distributions must retain the original copyright notice(s) in the source code, and also reproduce a copy of the license with every copy of the source code distributed. We are currently meeting Section 2.1(a) (keeping license notices in any original Watcom source files), but aren't reproducing an original copy of the OpenWatcom license.txt in ELKS. I will copy it over to libc/watcom/license.txt to meet this obligation, and thus should be meeting the license obligations.
The ELKS applications have a statement to that effect in elkscmd/LICENSE. Is there another file that should be modified to show that the ELKS C Library may include code that would be licensed under OpenWatcom license if used? |
Hello @Vutshi,
I hadn't thought of it quite like that, but after all your messing around with Intel vs AT&T syntax, and my having to switch to Intel syntax for the OpenWatcom
I think that it does, as there are some Another cool thing about the code OpenWatcom produces is that ES and DS REGISTERS ARE SCRATCH between function calls. That means that, especially in large/compact models, the compiler emits code to use DS and ES for quick and easy far pointer manipulation, and then uses SS: overrides for quick access to the application data segment. A bit hard to describe without showing lots of code, but quite a bit different than ia16-elf-gcc output. It seems that ia16-elf-gcc code generation was designed originally around small data model, while OpenWatcom has a much more complex register-allocation engine internally. Frankly, I hadn't ever considered writing ASM in the style of some of the code that is being output by OWC. I'm interested in profiling too. If you can learn more about it, please post what you find. |
Hello @ghaerr , Thank you for the clarification! |
I will if I eventually do something in that direction. Currently, I am obsessed with statically recompiling an old DOS game for ELKS and macOS ) |
Initial work for compilation and user testing of ELKS programs is now completed. This PR adds two scripts to more easily compile and link programs with OpenWatcom C for testing under ELKS. These are To test, set the WATCOM= and WATDIR= environment variables in a file, say wcenv.sh:
I have got the ELKS BASIC interpreter running (in compact model only for now). To compile and test yourself:
|
That's quite a feat! Maybe explaining the:
are suitable for a wikipage. |
Hello @ghaerr , Wow. https://github.com/tyama501/sjis-to-utf8_elks/blob/main/sjis_to_utf8.c By using the compact model, I know the kernel loader need to be enhanced to use data segment > 64KB, Thank you! |
Hello @tyama501,
Yes, having little code a lots of data is a good fit for compact model. However, far data (ie. using compact or large model) brings about its own complexities. One of them is in
Unfortunately, at the moment, no. We're still under heavy development, so this this may change. But for now, the ELKS C library In the future, I hope to rethink this a bit, but its complicated. Applications will still want the ability to allocate small amounts of memory from the heap, while also allocating memory outside the process. The Thus, compiling a 16-bit program in large model isn't just like flat 32-bit address space where one doesn't have to worry about memory.
BTW, OpenWatcom supports a "huge" memory model which will do that pointer arithmetic past 64K for you. I haven't even looked at it yet. Someone would need to read up on it in the Watcom Users's Manual. Having said all this, it could still be worthwhile to try compiling your program with OWC. Expect issues. For instance, I have ported only about 8 of the 35 system calls required; fmemalloc has not been ported. The good news is they are straightforward and you can add them yourself (or I will) in an easy PR.
Right now, both large and compact model programs should run with OWC as long as their code and data segments are < 64K, without waiting for the kernel enhancement. I am hoping to see you and others try out the compiler and we work through the issues before a kernel enhancement, so that we know more of what is needed. I plan to commit this PR today so you can try playing with the compiler. Thank you! |
Hello @toncho11,
Good idea. We're still in heavy development mode and the scripts will likely change. Agreed more information on how to compile up OpenWatcom from source is needed ASAP. I posted that information somewhere but will have to look it up. Perhaps you might try pulling down the repo and building it, I've only done this on macOS. Thank you! |
Hello @tyama501, (cross posting here from #1505 (reply in thread).
Large model should be quite easy to make happen, right now. The problem is that I haven't yet added support for both compact and large models simultaneously. Since the Watcom libc.lib is built in whole using In libc/watcom.inc:
In elks/tools/objtools/ewcc:
In elks/tools/objtools/ewlink make the same change as above. I suppose it might be better to change my scripts to large model immediately so that we have the most benefit. I had started with compact because it lessened the issues involved in the os2toelks executable format conversion but I think they are soled now. |
Thank you. And can we still use __far prefix? Or is this mean we don't need to use it?
At least we still need to do some kind of this unless we don't use the huge model right?
|
Hello @ghaerr , Before modifying for the fmemalloc, It seems that the compiler is complaining about char **argv cannot be used in the watcom? |
Long story short, declare main as follows:
|
Looking good. Now add the |
Hello @ghaerr , I could compile and link the sjis_to_utf8. I executed on ELKS and it got file memory too large. Previously HEAP was 65535 with gcc-ia16 and medium model. Thank you. |
Hello @tyama501,
Nice! System calls are pretty tightly coded with OWC and register calling sequences, aren't they :)
Yes, probably, but hard to tell at the moment. You might try re-compiling everything in compact model to see whether that makes much difference. (If EMoria already requires medium model then the ELKS kernel loader probably won't yet work with OWC-compiled programs because I haven't written any special code to convert OS/2 binaries with multiple code segments to ELKS a.out headers which have the 2nd code segment in .ftext (far text). So far, only single-code and single-data segments will be loadable by ELKS). The other reason I haven't written the conversion for multiple code segment programs is I haven't actually seen a large model OS/2 binary that is > 64k code, so not sure yet what will have to be done.
Yes, heap (and stack) are currently always being set to 0 (meaning default 4K default heap, 4K default stack) by the OS/2 binary converter
For now, you'll have to use You're welcome to submit a PR for fmemalloc.c :) Thank you! |
Hello @ghaerr ,
Yes, it seems that it uses ss and bp to get the pseg of fmemalloc. Hmmm. ![]() I will try compact model. Thank you. |
Hello @tyama501, I'm a bit confused about which versions of your program were originally small or medium model. But looking at the screenshots it seems that the program is only a small amount of code, but needs large data. Thus, the original ia16-elf-gcc program should probably be small model, and the OWC program should probably be compact model (near code, far data). This keeps our debugging to a minimum. With regards to stack and heap usage, both should be set to the previous values used in the ia16 version. However, it seems that what you're saying is that the ia16 vs OWC versions have different data segment sizes (use That said, it is not a good idea to arbitrarily change the stack size of a program without manually/visually verifying the stack requirements by inspecting the auto (stack) variables declared within its various functions. Stack overwrite problems on an 8086 can produce hard-to-find bugs. With regards to the heap: in small both small and compact/large models, as mentioned before, the Thus, the heap can be considerably lower than 65535 which just suggests max heap. Perhaps set the heap to 2-3k in your original ia16 program to ensure it works, then set the same in OWC. The idea here is to get to a program that has a small enough static data segment so that it can load and operate in ia16 and OWC. If the program data has many pointers, then those will move from 2 to 4 bytes and could easily become too large. In those cases, the program data segment needs to be made smaller, even in large model. Finally, I have identified a likely bug in Thank you! |
Thank you @ghaerr , I will read your explanation carefully later, It uses a lot of static memory because it has all sjis utf8 table inside. I guess it didn't work with the OWC because of of the fmemalloc bug. |
Ok, that's good to know. If it uses
Yes, that's what I thought. If there are few pointers, then the data segment size shown by
That could be, but also there are many other reasons, as there is very little testing a lots of code having been written for the OWC compilation. So we will have to be careful to understand exactly what might be going on. Another thought would be to test Now that I think more, perhaps use small model OWC also in the start, since this was also used by ia16 and worked. That will eliminate many other issues while we get things sorted out. |
〉You might test this with ia16 to determine whether a small heap value (1-2K) works. One question. It uses fopen. |
No - the
You will see that |
Hello @tyama501, I did more testing using the ELKS C stdio library compiled with Watcom for large model by compiling and linking elkscmd/misc_utils/hd.c and it seems to work well (large model). So I am not sure what the issue(s) might be with your SJIS to UTF-8 converter, other than perhaps we have a problem with large code segments, since hd.c is pretty small. But stdio seems to be working OK in large model from my testing. I am using the standard 4K stack and 4K heap. Thank you. |
Hello @ghaerr , Thank you. |
(I don't know why it needs so much STACK) |
I would think 2K heap will be needed at a minimum, certainly enough to remove the |
BTW, #1920 will require that you rewrite |
Interesting, although the sys_brk fail message could be coming at any time. Perhaps you should try running without output redirected so that we can more accurately determine when this is failing, e.g. Another idea is to insert write statements for debugging like I would have to see the source code to determine how much heap or stack are likely required. Can you post it? This isn't a large program, only a large data segment, right? Can approx 4-8k of data be remove from the program, just for testing purposes to allow using a 4K heap/4K stack? I will write a test OWC program using almost all of the data segment (by allocating static buffers) that also uses stdio and see if I can get a failure... |
Hello @tyama501, I tested stdio using a large model program whose data segment was almost completely full (134 bytes short of 65K including 4K stack and 4K heap, as space for environment and argv array are required). I could not get it to fail, so perhaps there is some other problem in your SJIS conversion program, I am not sure yet what it might be. Perhaps you can post source for your program or simplify it such that the brk_fail messages stop? Source for data.c:
compiled with:
Thank you! |
So after compilation usage is:
and it will output UTF8 by converting the SJIS format contained in sjis.txt. Output will be to |
Right. |
Fixes
#include <stddef.h>
problem discussed in #1911.Removes C library dependencies on external
<stddef.h>
,<stdint.h>
and `<sys/types.h> headers, which ended up dragging in unwanted stuff from OpenWatcom C.Locally defines
size_t
,ssize_t
,int64_t
,intptr_t
anduintptr_t
instead of using<stddef.h>
or<stdint.h>
for small, medium, compact and large models for ia16-elf-gcc and OpenWatcom.Adds first round of ELKS system calls to ELKS OpenWatcom C library:
open
,close
,read
,write
,lseek
,brk
,sbrk
andioctl
which allow stdio functions to work.Ported
malloc
/free
functions using internal__near
data model for large/compact model compilations.Temporarily redeclared
char **environ
to bechar __near * __far *environ
to allow direct access to environment array in data segment without rewriting it.main
s use ofchar **argv
must also be declared similarly for the time being to allow access to argv array without rewriting it in large/compact models.Added
make image
andmake images
upper level Makefile targets to allow rebuilding single or all images without rebuilding C library or applications; useful for quickly adding files to image root directory for testing.Fixed up intended error return when size > 32K bytes passed to
malloc
orsetvbuf
.Add fix to possibly require application code segment fixups to os2toelks.c when converting relocation entries. Maybe not needed but seeing some problems when linking large programs like
basic
.