[kernel] Add fmemalloc sys call, fix fsck on 65M disks

[ghaerr]

Adds _fmemalloc system call to allow processes to allocate far memory for themselves. Memory is automatically freed at process exit.
Adds fmemalloc wrapper and fmemset C library routines.
Enhances fsck to use far memory (up to 64K), which now allows working on max sized (65M) hard disks.
Rearranges some multiply-included kernel header files.

Fixes fsck as requested in #1312.

[Mellvik]

You know, This is close to incredible. Not only did we go from impossible to 'done' in almost no time (fsck), we also got a system level enhancement that will benefit ELKS wide and far! Thank you @ghaerr, I'm really enthusiastic about this. If I understand it correctly, this is What we used to call upper memory, available on most system post-XT. I can't wait to see what it may do for commands like dd (with raw devices ... ), tar, compress, and more, Great work! Test results coming.

…

-M

10. jun. 2022 kl. 04:25 skrev Gregory Haerr ***@***.***>:  Adds _fmemalloc system call to allow processes to allocate far memory for themselves. Memory is automatically freed at process exit. Adds fmemalloc wrapper and fmemset C library routines. Enhances fsck to use far memory (up to 64K), which now allows working on max sized (65M) hard disks. Rearranges some multiply-included kernel header files. Fixes fsck as requested in #1312. You can view, comment on, or merge this pull request online at: #1314 Commit Summary 975c6d8 [kernel] Add fmemalloc sys call, fix fsck on 65M disks e215754 cleanup File Changes (26 files) M elks/arch/i86/drivers/char/console-bios.c (3) M elks/arch/i86/drivers/char/console-direct-pc98.c (3) M elks/arch/i86/drivers/char/console-direct.c (3) M elks/arch/i86/drivers/char/mem.c (1) M elks/arch/i86/drivers/char/ntty.c (1) M elks/arch/i86/drivers/net/ne2k.c (1) M elks/arch/i86/kernel/process.c (1) M elks/arch/i86/kernel/strace.h (1) M elks/arch/i86/kernel/syscall.dat (1) M elks/arch/i86/kernel/timer.c (1) M elks/arch/i86/mm/malloc.c (39) M elks/fs/file_table.c (1) M elks/fs/minix/symlink.c (1) M elks/fs/msdos/dir.c (1) M elks/include/linuxmt/mm.h (40) M elks/kernel/exit.c (4) M elks/kernel/printk.c (2) M elks/kernel/time.c (1) M elkscmd/disk_utils/fsck.c (23) M elkscmd/sys_utils/meminfo.c (2) M libc/include/malloc.h (3) M libc/include/string.h (3) M libc/malloc/Makefile (1) A libc/malloc/fmemalloc.c (17) M libc/string/Makefile (1) A libc/string/fmemset-c.c (15) Patch Links: https://github.com/jbruchon/elks/pull/1314.patch https://github.com/jbruchon/elks/pull/1314.diff — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.

[ghaerr]

This is close to incredible. Not only did we go from impossible to 'done' in almost no time (fsck)

Thanks, I had forgotten that I had previously thought about using far memory buffers for this the last time I looked. It turned out to be pretty straightforward after that.

If I understand it correctly, this is What we used to call upper memory, available on most system post-XT. I can't wait to see what it may do for commands like dd (with raw devices ... ), tar, compress, and more,

No - not quite. For the first version, I hardcoded 0xD000, which is the segment address of the upper memory portion of PC memory. However, this final version just allows any ELKS application program to ask for memory OUTSIDE it's own address space. The current implementation gets memory from the kernel memory manager, which allocates from main memory, which does not include upper memory. We have been talking about adding upper memory to the main memory allocation routine, and that would help for systems that have little memory, but other than that won't speed up application programs. This upper memory is also already usable from RAM disks on ELKS, set in config.

If you think we should add the capability to use upper memory, I'll add that to the list.

[ghaerr]

However, our new fmemalloc routine would allow for ELKS programs to allocate any amount of available (linear) main memory for their own use, allowing, for instance 64K - 256K buffers for dd. But, since the low-level disk I/O is only performed in 1K chunks, I don't really think this will help for any I/O bottlenecks. It does allow for programs with much larger data requirements to be written, which would use the char __far * type to access far data. Due to compiler limitations though, the data is only accessible in 64k segments, but that's not too hard to handle.

[Mellvik]

Ah, I didn't get that change from the 'faked up' to the 'real' implementation. So with the 'real' (current) implementation, the improvement is more memory available per process, while still inside the 640k range, right? That doesn't change the value and importance of the improvement. The opposite can be argued: The improvement is available to all, not only those who have UMBs available. ELKS can now do things that weren't possible before. Such as (possibly, I'm assuming this based on previous discussions) yanking compress to 16 bits from 12, even though it will be slow. Run kilo decently (maybe?). Run rz/sz serial comms programs... the list goes on. As to making upper memory available, yes, I think that wold be very valuable. If it could be folded into the fmemalloc framework you've already created, and possibly enabled via /bootopts we'd have a party. In particular if it could free up some low memory for the kernel so the # of processes can be increased. For example. With 3 serial lines and several network connections active (a regular situation while testing) I'm hitting the process ceiling quite frequently these days. Testing the new fsck this weekend. Thank you!

…

--M

10. jun. 2022 kl. 22:51 skrev Gregory Haerr ***@***.***>:  This is close to incredible. Not only did we go from impossible to 'done' in almost no time (fsck) Thanks, I had forgotten that I had previously thought about using far memory buffers for this the last time I looked. It turned out to be pretty straightforward after that. If I understand it correctly, this is What we used to call upper memory, available on most system post-XT. I can't wait to see what it may do for commands like dd (with raw devices ... ), tar, compress, and more, No - not quite. For the first version, I hardcoded 0xD000, which is the segment address of the upper memory portion of PC memory. However, this final version just allows any ELKS application program to ask for memory OUTSIDE it's own address space. The current implementation gets memory from the kernel memory manager, which allocates from main memory, which does not include upper memory. We have been talking about adding upper memory to the main memory allocation routine, and that would help for systems that have little memory, but other than that won't speed up application programs. This upper memory is also already usable from RAM disks on ELKS, set in config. If you think we should add the capability to use upper memory, I'll add that to the list. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

[ghaerr]

the improvement is more memory available per process, while still inside the 640k range, right?

Yes.

yanking compress to 16 bits from 12, even though it will be slow.

Yes, that could probably be done now by allocating a far (char __far *) buffer.

Run kilo decently (maybe?).

No, the problem with kilo is that it redraws the screen whenever an arrow key is pressed. I did realize we have mined (MINIX editor) ported to ELKS. It is MINIX's screen editor. @toncho11 will probably find it superior. Perhaps we should rename it 'edit' so that it is obvious what it is.

if it could free up some low memory for the kernel so the # of processes can be increased.

Unfortunately, fmemalloc won't free up any kernel data segment memory, which is tight. The max process boundary could be increased by allocating task structures dynamically, which could help you. There are some issues going that route, which we can discuss, but it might allow for 5-6 more processes or more, than the current 16 before kernel memory runs out again. Another issue you're likely having with 3 serial lines open is the large 1K receive buffer which is by default allocated to each. This comes out of kernel memory also, and ultimately would compete with the number of tasks available if dynamically allocated.

[Mellvik]

yanking compress to 16 bits from 12, even though it will be slow. Yes, that could probably be done now by allocating a far (char __far *) buffer.

Great that would be very convenient for compressed tar files. I always forget the tar option to set 12 bit compression and have to do it all over again.

Run kilo decently (maybe?). No, the problem with kilo is that it redraws the screen whenever an arrow key is pressed. I did realize we have mined (MINIX editor) ported to ELKS. It is MINIX's screen editor. @toncho11 <https://github.com/toncho11> will probably find it superior. Perhaps we should rename it 'edit' so that it is obvious what it is.

OK (kilo), Great idea (rename to 'edit'). Personally I'm more than content with vi, but it seems an alternative is needed ...

if it could free up some low memory for the kernel so the # of processes can be increased. Unfortunately, fmemalloc won't free up any kernel data segment memory, which is tight. The max process boundary could be increased by allocating task structures dynamically, which could help you. There are some issues going that route, which we can discuss, but it might allow for 5-6 more processes or more, than the current 16 before kernel memory runs out again.

5-6 more would be great - and sufficient!

Another issue you're likely having with 3 serial lines open is the large 1K receive buffer which is by default allocated to each. This comes out of kernel memory also, and ultimately would compete with the number of tasks available if dynamically allocated.

Makes sense. Now, if there is (dynamic) competition for the resource, the user (me in this case) would be able to prioritize: Ditch a serial line when more processes are required. Thank you! —M

[ghaerr]

I always forget the tar option to set 12 bit compression and have to do it all over again.

I'm not sure we want to automatically default to 16-bit compression for ELKS, especially because there may not always be memory for it.

Moving to 12-bit default (rather than a separate option for it), might be a good idea though.

What exactly is your issue, are you talking about running on ELKS, or host, and is it a tar option, or the compress -b 12 you forget? Perhaps these could be defaulted so that things work more as expected.

Now, if there is (dynamic) competition for the resource, the user (me in this case) would be able to prioritize: Ditch a serial line when more processes are required.

Exactly. The further issue is that since the kernel would be allocating a relatively large structure from the heap (~800 bytes for each task struct, 1K bytes for serial), when/if the kernel heap gets fragmented, we could potentially permanently lose the ability to allocate the required structure. This should only be an issue with a lot of longer-running processes though.

[Mellvik]

I always forget the tar option to set 12 bit compression and have to do it all over again. I'm not sure we want to automatically default to 16-bit compression for ELKS, especially because there may not always be memory for it. Moving to 12-bit default (rather than a separate option for it), might be a good idea though. What exactly is your issue, are you talking about running on ELKS, or host, and is it a tar option, or the compress -b 12 you forget? Perhaps these could be defaulted so that things work more as expected.

I agree, and I didn't mean changing the default on ELKS. The problem is always TO ELKS. When untaring, tar will use the right bit length automatically if supported by the host system. —M

[ghaerr]

The problem is always TO ELKS. When untaring, tar will use the right bit length automatically if supported by the host system.

I see. So the ability for compress -d on ELKS to decompress a 16-bit compressed file is quite useful. I'll add that to my list to see how compress on ELKS might be enhanced using fmemalloc. Thanks!

[Mellvik]

Thanks @ghaerr - and btw — This is looking good: elks15# fsck -r //dev/hda4 Filesystem on //dev/hda4 is dirty, needs checking. elks15# mount /dev/hda4 /mnt4 elks15# mount /dev/fd0 (minix) blocks 1440 free 395 mount / /dev/hda4 (minix) blocks 64000 free 62390 mount /mnt4 elks15# —M

…

11. jun. 2022 kl. 18:44 skrev Gregory Haerr ***@***.***>: The problem is always TO ELKS. When untaring, tar will use the right bit length automatically if supported by the host system. I see. So the ability for compress -d on ELKS to decompress a 16-bit compressed file is quite useful. I'll add that to my list to see how compress on ELKS might be enhanced using fmemalloc. Thanks! — Reply to this email directly, view it on GitHub <#1314 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA3WGOGMFSTDWYU7FH6I4ZDVOS673ANCNFSM5YMELORQ>. You are receiving this because you commented.

[Mellvik]

One more thing re. fsck @ghaerr. I've been consistently using the -a option (and -r) with fsck, and some times been suprised by it not working. Now that I can finally check a full 64M fs that has been used for a while, the 'non-automatic' behaviour became annoying (hundreds of errors). Surprisingly (I haven't looked at the source before), it turns out that -a is negated by -r, which explains what I thought was erratic behaviour. -ra is different from -ar because -r turns off -a. I suggest this negation be removed from -r. Whether -a should continue to imply -r is a different story. Ideally it should not, but option processing becomes easier that way (since -a alone is not meaningful otherwise). —M

…

11. jun. 2022 kl. 20:28 skrev Helge Skrivervik ***@***.***>: Thanks @ghaerr - and btw — This is looking good: elks15# fsck -r //dev/hda4 Filesystem on //dev/hda4 is dirty, needs checking. elks15# mount /dev/hda4 /mnt4 elks15# mount /dev/fd0 (minix) blocks 1440 free 395 mount / /dev/hda4 (minix) blocks 64000 free 62390 mount /mnt4 elks15# —M > 11. jun. 2022 kl. 18:44 skrev Gregory Haerr ***@***.*** ***@***.***>>: > > > The problem is always TO ELKS. When untaring, tar will use the right bit length automatically if supported by the host system. > > I see. So the ability for compress -d on ELKS to decompress a 16-bit compressed file is quite useful. I'll add that to my list to see how compress on ELKS might be enhanced using fmemalloc. Thanks! > > — > Reply to this email directly, view it on GitHub <#1314 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA3WGOGMFSTDWYU7FH6I4ZDVOS673ANCNFSM5YMELORQ>. > You are receiving this because you commented. >

[ghaerr]

it turns out that -a is negated by -r, which explains what I thought was erratic behaviour. -ra is different from -ar because -r turns off -a.
I suggest this negation be removed from -r.

Ok - quite confusing, since -ra is not the same as -ar.

Whether -a should continue to imply -r is a different story. Ideally it should not, but option processing becomes easier that way

I think -a should be left alone - it appears that the original intent was to either run fsck with -a or with -r, and some folks may continue to think that way, which will still work.

Thank you!