[cmds] Add Kilomacs (Rafael's own version of emacs :P ) editor

[rafael2k]

After spending countless hours trying to make jove run perfectly... I decided to just adapt kilo to use emacs key bindings. I improved memory handling too, so I can edit drawcube.lua comfortably in ELKS, without memory exhaustion.

It adopts the default "emacs" naming, like "elvis". Let me know if this is ok.

[ghaerr]

Having a name change after make via mv isn't really a good idea here. Can you either change the Makefile in kilomacs or perform this filename change in ExtApplications?

[rafael2k]

I'll change in the Makefile

[ghaerr]

Thanks @rafael2k!

[rafael2k]

ELKS native development environment keeping evolving!

[ghaerr]

ELKS native development environment keeping evolving!

Yes, this is looking very nice, with multiple editors and a working devkit, all automatically distributed.

I saw you added mem.c to kilo to allow for larger files to be edited. Have you learned what a typical max file size is?
Also, is there a way to show kilomacs/emacs key bindings? I'm not that familiar with emacs and am wondering how different the bindings are than the "stock" kilo.

[rafael2k]

I added a help, which can be accessed with Ctrl + [
I'll test the limits of kilo and write here.

[rafael2k]

I can open stdio.h (5307 bytes), but it fails with a > 10kB file. I think I did something wrong. Lemme check why a 10kB+ file is not opening fine.

(23)SBRK 10244 FAIL, OUT OF HEAP SPACE

Insufficient memory to load file reading line 256!

[rafael2k]

One question: strdup is also covered by our "mem.c"?
Somehow memory seems to still being allocated in the heap.

[ghaerr]

You can always look at the source in libc/string/strdup.c, it calls malloc. Remember that mem.c allocates small strings less than MALLOC_ARENA_THRESH to the local heap. Kilo could be allocating each line separately, in which case they're all going into the local heap. You'll have to chase that down in kilo.c. Alternatively you could rewrite mem.c to always allocate from the far heap and/or set the threshold to 0 or very small.

[rafael2k]

I had to fix some bugs, there was some outstanding memory issues that was causing some random crashes. At least the code is small...

So when I link the software with owc, even if strdup calls malloc (and both of them are in the libc), the linker will still "prefer" the malloc in the mem.c, correct? No malloc is "escaping" to the default libc implementation, right?

ps: playing with arena threshold and even using just the far memory, I already managed to open 20kB files - Things are not that bad! As soon as I understand more about kilo internals I'll set the final memory tunables.

ps2: I can set the heap in owc very low right (-Wl,heapsize), since it will not be used, correct?

[ghaerr]

So when I link the software with owc, even if strdup calls malloc (and both of them are in the libc), the linker will still "prefer" the malloc in the mem.c, correct? No malloc is "escaping" to the default libc implementation, right?

Yes, correct. Since the symbol malloc is seen in the program, any library version of the same name will not be pulled in or linked.

playing with arena threshold and even using just the far memory, I already managed to open 20kB files - Things are not that bad!

Right. You'll need to best understand the sizes and frequency of malloc in order to best tune the program. For instance, even though all mallocs could come from far memory, each one actually consumes a paragraph (16 bytes) of memory for even a single byte allocation - expensive. Also, main memory could get checkerboarded with long-lived allocations which may prevent later large allocations, etc.

I can set the heap in owc very low right (-Wl,heapsize), since it will not be used, correct?

Yes. In general for this type of mem.c configuration, it can be set to the size of allocations expected in the local heap, or just 256 or 512 if mem.c won't ever allocate from it.

[rafael2k]

With -Wl,heapsize=0x200 the software does not even open, and I already get:
SBRK: 178 FAIL, OUT OF HEAP SPACE

The space allocated for the arena is not in the heap (this one set in OWC cmd line), right?
I got a bit lost.

[ghaerr]

SBRK: 178 FAIL, OUT OF HEAP SPACE

Actually there are other library routines that may try to allocate from the local heap, like exec(). This allocation shows 178 bytes. We would have to research the source code to see what it is doing to learn more. These use sbrk in order to get some memory. So for now, use the standard heap size of 0x1000 (4K) and continue debugging your mem.c. Most of the allocation should be sent to far memory, by reducing MALLOC_ARENA_THRESH as discussed above.

[rafael2k]

Something is really allocating memory from the "default" heap. May be calling sbrk() directly? With 1k heap not even drawcube.lua opens (SBRK 58 FAIL, ...)

[ghaerr]

The space allocated for the arena is not in the heap (this one set in OWC cmd line), right?

Yes, that's correct. Mem.c uses a single far heap, allocated once using MALLOC_ARENA_SIZE bytes max (allocated from main memory), and the rest using individual fmemalloc calls out of main memory. It is possible you're running out of memory from the _fmalloc() call in mem.c, add a __dprintf to check. The _fmalloc allocator uses only 4 bytes overhead per allocation, while fmemalloc uses 16. Debug statements in mem.c will help, and set the near heap size to 1K or 4K and the program should be good.

[ghaerr]

Something is really allocating memory from the "default" heap. May be calling sbrk() directly? With 1k heap not even drawcube.lua opens (SBRK 58 FAIL, ...)

I thought you're talking about kilomacs. If you're talking about Lua, that could be far more complicated, I have no idea how it is allocating memory. You have to understand the memory allocation mechanism(s) the program is using in order to code mem.c properly.

Set the default near heap to 60k and proceed. That should allow you to find out how large a file kilomacs can edit.

[rafael2k]

It is kilomacs. I disabled a "open recent files" feature. Lets see.

ps: I'm using drawcube.lua just as a test text file for kilomacs!
: )

[ghaerr]

I'm using drawcube.lua just as a test text file for kilomacs!

I get it now, sorry. Set the local heap to 60k, and then add debug statements to mem.c. That will allow you to use a max local heap and also see which malloc allocations are being routed through the two arenas it supports. If you display the size values and use __dprintf to display to the debug console rather than direct console, you will be able to get a good idea of how many and how big each allocation is.

[rafael2k]

trying to open a big file, I get an error from SBRK too
SBRK 10244 FAIL

I need to debug who is calling sbrk().

[ghaerr]

I need to debug who is calling sbrk().

Yes, perhaps kilo itself doesn't call malloc but uses sbrk for the entire size of the file? If so, then you'll need to rewrite that.

[ghaerr]

SBRK 10244 FAIL

sbrk should work if the local heap is small, say 4k. This is because initially the heap is unused, and sbrk extends the "top" of it. So you might try running with a smaller initial heap, and let kilo use sbrk to extend it. If sbrk is used for allocation, then your mem.c won't do much good, as only internal C library routines would be calling it. Look for explicit calls to malloc in kilo.c to find out.

[rafael2k]

Uff, I think I forgot to define ELKS for mem.c

[rafael2k]

I broke the 20k barrier:

Memory handling in kilo is not that good... I wonder the speed of it in a real XT...

[rafael2k]

All good, sorry the noise...
It was good I understood a bit about kilo internals, and fixed a couple of bugs.
I tried to print all malloc-type allocations - it is madness - sooo many mallocs and frees.

[ghaerr]

All good, sorry the noise...

What is the approximate max size of file that can be edited with kilomacs now?

[rafael2k]

With stability, about 20kB (a bit more, but not much more), which is much more than the default kilo, which can not even open stdio.h with its 5kB.