Console Web Browser

[rafael2k]

I started trying to compile bobcat (a lynx fork) for ELKS. While I'll have some work to remove all unneeded stuff to fit in the executable size constraints, here is the current status of trying to link all the mess (I'm still using a placeholder curses which I borrowed from tui):

owcc -bos2 -s -Wl,option -Wl,dosseg -Wl,option -Wl,start=_start -Wl,option -Wl,nodefaultlibs -Wl,option -Wl,stack=0x2000 -Wl,option -Wl,heapsize=512 -o bobcat DefaultStyle.obj HTAlert.obj LYDownlo.obj LYHistor.obj LYLeaks.obj LYMainLoop.obj LYrcFile.obj LYShowIn.obj GridText.obj HTML.obj LYClean.obj LYEdit.obj LYJump.obj LYOption.obj LYReadCF.obj LYString.obj LYBookmark.obj LYCurses.obj LYGetFil.obj LYKeymap.obj LYMain.obj LYPrint.obj LYSearch.obj LYUtils.obj LYEditMap.obj HTWrite.obj HTForms.obj lib/HTAccess.obj lib/HTAuth.obj lib/HTLex.obj lib/HTParse.obj lib/HTStyle.obj lib/HTBTree.obj lib/HTFWrite.obj lib/HTList.obj lib/HTPasswd.obj lib/HTTCP.obj lib/SGML.obj lib/HTAnchor.obj lib/HTChunk.obj lib/HTMIME.obj lib/HTPlain.obj lib/HTTP.obj lib/HTAssoc.obj lib/HTFile.obj lib/HTMLDTD.obj lib/HTUU.obj lib/HTAtom.obj lib/HTFormat.obj lib/HTInit.obj lib/HTMLGen.obj lib/HTString.obj lib/HTWriter.obj lib/HTGroup.obj lib/HTACL.obj lib/HTAAServ.obj lib/HTAAProt.obj lib/HTAAFile.obj lib/HTAABrow.obj lib/HTRules.obj lib/HTAAUtil.obj curses/curses2.obj curses/curses3.obj curses/curses.obj curses/unikey.obj curses/tty.obj mem.obj -Wl,library -Wl,/home/rafael2k/programs/devel/elks/libc/libc.lib
Error! E2028: socket_ is an undefined reference
Error! E2028: connect_ is an undefined reference
Error! E2028: bind_ is an undefined reference
Error! E2028: alarm_ is an undefined reference
Error! E2021: size of segment _BSS exceeds 64k by 11656 bytes
Error! E2021: size of segment STACK exceeds 64k by 11664 bytes
Error! E2021: size of segment STACK exceeds 64k by 11664 bytes
Error! E2021: size of segment STACK exceeds 64k by 19856 bytes
Error: Linker returned a bad status
make: *** [Makefile.elks:62: bobcat] Error 1

Lynx is massive. I did not know that.

I opened an issue to discuss some missing functions here: #2253

This is my port repo: https://github.com/rafael2k/bobcat/

[ghaerr]

I'm still using a placeholder curses which I borrowed from tui

Yes, your initial post showed lots of unimplemented curses functions. You'll want to check that Lynx/Bobcat doesn't need to use the "window" functions in curses, which actually implement overlapping windows, etc - this feature is not implemented at all and would take some work. Most curses programs don't need the "internal windowing" features of curses and just use the cursor positioning etc stuff. If you need to add that, for instance for winsertln, just add the ANSI console sequence for that function. (None of this will work if the insert line, clear window, etc need to operate within a curses "window" rather than full screen. It would be worth looking into before you spend too much time on the port.

Error! E2028: socket_ is an undefined reference
Error! E2028: connect_ is an undefined reference
Error! E2028: bind_ is an undefined reference
Error! E2028: alarm_ is an undefined reference

As indicated in #2253, these system calls are missing; I will add them.

Error! E2021: size of segment _BSS exceeds 64k by 11656 bytes

Looks like you'll need to use large model. If you already are, then you've got some major porting issues, and you'll have to programmatically determine which global/static data structures should be placed in a separate segment. Read up on OWC -ztN (max default data threshold) option for more details on one way to do that.

[rafael2k]

I'm already in the large mode. Bobcat uses window just for the forms, which I can kind of hack around.
I'll read on how to specify where data structures go. I'll have a go on it, I already spend most of the week on bobcat, and can spend a couple more days to see where I can go.

[rafael2k]

Already manage to reach here:

owcc -bos2 -s -Wl,option -Wl,dosseg -Wl,option -Wl,start=_start -Wl,option -Wl,nodefaultlibs -Wl,option -Wl,stack=0x2000 -Wl,option -Wl,heapsize=512 -o bobcat DefaultStyle.obj HTAlert.obj LYDownlo.obj LYHistor.obj LYLeaks.obj LYMainLoop.obj LYrcFile.obj LYShowIn.obj GridText.obj HTML.obj LYClean.obj LYEdit.obj LYJump.obj LYOption.obj LYReadCF.obj LYString.obj LYBookmark.obj LYCurses.obj LYGetFil.obj LYKeymap.obj LYMain.obj LYPrint.obj LYSearch.obj LYUtils.obj LYEditMap.obj HTWrite.obj HTForms.obj curses/curses2.obj curses/curses3.obj curses/curses.obj curses/unikey.obj curses/tty.obj mem.obj lib/SGML.obj lib/HTAnchor.obj lib/HTChunk.obj lib/HTMIME.obj lib/HTPlain.obj lib/HTTP.obj lib/HTList.obj lib/HTAccess.obj lib/HTAuth.obj lib/HTLex.obj lib/HTParse.obj lib/HTStyle.obj lib/HTBTree.obj lib/HTFWrite.obj lib/HTPasswd.obj lib/HTTCP.obj lib/HTAssoc.obj lib/HTFile.obj lib/HTMLDTD.obj lib/HTUU.obj lib/HTAtom.obj lib/HTFormat.obj lib/HTInit.obj lib/HTMLGen.obj lib/HTString.obj lib/HTWriter.obj lib/HTGroup.obj lib/HTACL.obj lib/HTAAServ.obj lib/HTAAProt.obj lib/HTAAFile.obj lib/HTAABrow.obj lib/HTRules.obj lib/HTAAUtil.obj -Wl,library -Wl,/home/rafael2k/programs/devel/elks/libc/libc.lib
Error! E2021: size of segment _BSS exceeds 64k by 9520 bytes
Error! E2021: size of segment STACK exceeds 64k by 9520 bytes
Error! E2021: size of segment STACK exceeds 64k by 17712 bytes
Error: Linker returned a bad status
make: *** [Makefile.elks:61: bobcat] Error 1

Which manual are you using for getting references like the "-ztN"?

Btw, which codepage does ELKS use?
I'm questioning on Latin1 or codepage 437, as I'm removing all large charset definitions to save up some space on "LYCharSets.c".

by changing the buffer[] allocations to dynamic memory allocations would reduce the stack usage, right? The BSS is uninitialized data, correct?
Trying to picture what can be done here.

ps: about curses - I ifdef'd many window functions in HTForms just to compile the code first, as a way to check the feasibility of the project first, before diving in the (not so big) use of curses by bobcat.

[rafael2k]

-Wc,-zt512 made the deal! Yay!

now the work continues to clean up all the DOS cruft and see if it can connect over http.

[rafael2k]

It tried to open a website, but I think it crashed ELKS somehow. I started seeing this:

fmalloc assert fail: testbusy(next(p)) (line 217)

After the first time I run bobcat, second time it does not load anymore (and I see in console the error above).

[rafael2k]

But with ktcp I get not enough memory.
:/

[ghaerr]

fmalloc assert fail: testbusy(next(p)) (line 217)
fmalloc assert fail: FP_SEG(ptr)==allocseg (line 213)

These messages mean that the fmalloc heap is corrupted. This could be as a result of -zt512 data threshold changing where data is allocated, but more likely other reasons - memory being scribbled on from other issues.

with ktcp I get not enough memory.

Browsers are well known for firing off many TCP connection requests simultaneously. I hadn't thought of this, but ELKS is quite limited in the number of simultaneous network connections, as each on allocates from ktcp's heap, possibly as high as 2-4k each IiRC). Try increasing ktcp's heap to max, its set to 32k now. Also see if Bobcat can be told to limit the number of simultaneous connections.

After the first time I run bobcat, second time it does not load anymore (and I see in console the error above).
Also this error after first run (in the second run):

Whenever memory corruption occurs, it is not safe to continue using ELKS for any reason. I suggest rebooting, since there is no hardware kernel or other application (e.g. shell) protection, and anything may have been overwritten.

[rafael2k]

Also this error after first run (in the second run):

fmalloc assert fail: FP_SEG(ptr)==allocseg (line 213)

[ghaerr]

Btw, which codepage does ELKS use?

ELKS doesn't use any codepage, but depends on the code page of the hardware character ROM, which on IBM PC is usually 437 in US. It may be slightly different with some Euro PCs. The console drivers never convert the byte stream nor decode UTF-8, etc.

by changing the buffer[] allocations to dynamic memory allocations would reduce the stack usage, right? The BSS is uninitialized data, correct?

BSS data is any uninitialized global or static declaration. When using -zt512, any .data or .bss declaration larger than 512 will be placed in a separate data segment, that is, a reload of the DS register will be generated in order to access it. Thus there are far pointers being used throughout the program. Otherwise, all data access would use the initial DS value, get it? The stack is always referenced from the "default" data segment using SS, and initially DS==SS.

If buffer[] allocations are from the stack, you'll definitely need to increase stack size, or stack overflow will happen, possibly producing your errors above. Dynamic allocation would also help; you should look at the sizes of these allocations to understand what to do. Our _fmalloc implementation is still single-heap, so the program is still limited to a max 64k unless you bring in mem.c etc like we did with 8086 toolchain.

[rafael2k]

Btw, which codepage does ELKS use?

ELKS doesn't use any codepage, but depends on the code page of the hardware character ROM, which on IBM PC is usually 437 in US. It may be slightly different with some Euro PCs. The console drivers never convert the byte stream nor decode UTF-8, etc.

ok

by changing the buffer[] allocations to dynamic memory allocations would reduce the stack usage, right? The BSS is uninitialized data, correct?

BSS data is any uninitialized global or static declaration. When using -zt512, any .data or .bss declaration larger than 512 will be placed in a separate data segment, that is, a reload of the DS register will be generated in order to access it. Thus there are far pointers being used throughout the program. Otherwise, all data access would use the initial DS value, get it? The stack is always referenced from the "default" data segment using SS, and initially DS==SS.

Got it. I noticed indeed the code gets larger because of this.

If buffer[] allocations are from the stack, you'll definitely need to increase stack size, or stack overflow will happen, possibly producing your errors above. Dynamic allocation would also help; you should look at the sizes of these allocations to understand what to do. Our _fmalloc implementation is still single-heap, so the program is still limited to a max 64k unless you bring in mem.c etc like we did with 8086 toolchain.

I already integrated mem.c.

btw, bobcat does not load load when ktcp is on:

./bobcat: not enough memory

[ghaerr]

bobcat does not load load when ktcp is on:

Well, now we're getting close to the absolute limits of what can be done on ELKS with its limited 1M addressing and only 640K of RAM!!! Do you have UMB support? Enable it. Etc.

Is this with the changed ktcp w/additional 32k heap? If so, this means you're fighting with literally +/- 32k ram. You'll want to run meminfo and take a hard look. Login as toor and run sash instead of sh. Possibly use the /bootopts #task=6 buf=8 cache=4 file=20 inode=24 heap=15000 n # min w/no rc.sys line enabled (with much playing around with kernel heap requirements). Run bobcat only immediately after boot for no memory fragmentation, etc.

[rafael2k]

I did not change ktcp. I'm using stock one. Will try the other options!

[toncho11]

@rafael2k Here is the memory increase page:
https://github.com/ghaerr/elks/wiki/Increase-available-memory

[rafael2k]

But even without network yet, I could render a .html from local, after changing some buffers (but just once, second time I get the "fmalloc assert fail: testbusy(next(p)) (line 217)"):

So far, I thought it would not be possible at all. Now I have even a small bit of hope things might work. There is still some cruft to cleanup.

[rafael2k]

After a lot of fuzzing I managed to run bobcat with ktcp. Many issues to solve still, but what puzzles me is ELKS state after running bobcat. All the other software seems to run fine, apart of bobcat itself.

[ghaerr]

what puzzles me is ELKS state after running bobcat.

I'm not sure I understand the problem - are you saying that after running bobcat, ELKS no longer runs well? That would likely be memory corruption as described above.

After a lot of fuzzing I managed to run bobcat with ktcp.

What do you mean, that you got bobcat to connect via http and retrieve a webpage?

[rafael2k]

what puzzles me is ELKS state after running bobcat.

I'm not sure I understand the problem - are you saying that after running bobcat, ELKS no longer runs well? That would likely be memory corruption as described above.

ELKS seems to keep running well, but bobcat would not run anymore, with errors like:

    fmalloc assert fail: testbusy(next(p)) (line 217)
    fmalloc assert fail: FP_SEG(ptr)==allocseg (line 213)

After a lot of fuzzing I managed to run bobcat with ktcp.

What do you mean, that you got bobcat to connect via http and retrieve a webpage?

That is the next step. It tries to connect and exits, prolly I forgot something in HTTP.c.

[ghaerr]

Check you've got enough stack, although the kernel checks that on each system call (but not in between).

Perhaps add printf debug statements between allocations to try to find when the fmalloc assert is happening.

You could live dangerously and remove the abort() in libc/malloc/fmalloc.c to allow the program to continue operating. (Not recommended except for debugging!)

[rafael2k]

Check you've got enough stack, although the kernel checks that on each system call (but not in between).

Perhaps add printf debug statements between allocations to try to find when the fmalloc assert is happening.

You could live dangerously and remove the abort() in libc/malloc/fmalloc.c to allow the program to continue operating. (Not recommended except for debugging!)

Btw, the issue was a massive memory corruption I introduced when porting the DOS code to ELKS/Linux.

[rafael2k]

With a little more work on the network side, I think the ELKS port will be ready, but it does not mean it will run (because of size constraints). I'm testing on linux right now, and the tui curses is doing a good job - I can even type in a search box of frogfind.com and go to search, and it works (of course the form filling is visually broken and characters appear in the wrong place, but it actually works correctly).

[toncho11]

There is an http version of frogfind.com? It says "Server error" when I try "http://frogfind.com".

[rafael2k]

There is an http version of frogfind.com? It says "Server error" when I try "http://frogfind.com".

http://www.frogfind.com/

[rafael2k]

I made release "pre-alpha" of this ancient Lynx for Elks (reading the source code, it is really a mashup of lynx ~2.4 with 2.5 and 2.6, with the libwww by Tim Berners-Lee et al.):
https://github.com/rafael2k/bobcat/releases/tag/0.1

[rafael2k]

For local html it is running ok. For http with ktcp running... ktcp crashes.
: )

[rafael2k]

This is a capture of bobcat displaying the previously downloaded html of frogfind.com:

ps: on ELKS!

[rafael2k]

Current specs:

[ghaerr]

ps: on ELKS!

Pretty cool! :)

For local html it is running ok. For http with ktcp running... ktcp crashes.

Make sure to increase the heap for ktcp in ktcp/Makefile to 0xffff, even though it looks like the data segment is already pretty high at 57k. I would also suggest using __dprintf around some of the socket or connection requests in the Lynx code to display on the console when a connection is attempted. In this way you can possibly count connections or otherwise learn more about what is going on. I would guess that ktcp is running out of memory; there may be cases where a NULL malloc return isn't checked. We might also be able to configure ktcp to use less memory per socket for the incoming buffer. That would likely involved shortening the TCP segments as well.

[rafael2k]

Yay!

Definitely - I'll investigate better, add some __dprintfs and increase the heap for ktcp.
Never really played with tcp/ip on ELKS.

[toncho11]

If memory is not enough try:

https://github.com/ghaerr/elks/wiki/Increase-available-memory
https://github.com/ghaerr/elks/wiki/Access-to-UMB-memory

You can use all the video memory in this case for RAM. But it is tricky. You need to know what is loaded where for each computer and emulator to be sure it won't crash. So for UMB you need to use https://winworldpc.com/product/microsoft-diagnostic/5x and explore in DOS the memory between 640 and 1MB. Then write this down and configure it in ELKS UMB configuration in /bootopts.

[rafael2k]

Thanks. I also read about XMS remapping to UMB, as you mentioned.

But I'm cautiously optimistic we'll be able to run bobcat and ktcp in a stock ELKS without extra RAM. Lets see.

[rafael2k]

Good news - ktcp is not crashing! bobcat is locking in some call, I'll investigate.

[rafael2k]

And, after copying the urlget code, I can browse the nineties web! (at least once, then second time system freezes, which is reproducible also with urlget):

[rafael2k]

In order to have more free ram in ELKS, to run bobcat I enable network,
but disable ftpd and httpd, and add this to /bootops:

task=12 buf=8 cache=4 file=12 inode=16 heap=24000

[toncho11]

Man servers expect HTTP/1.1 behavior and may return responses using features (like persistent connections or chunked encoding) that the client isn’t prepared to handle. Although most servers can still process HTTP/1.0 requests, relying on HTTP/1.0 might lead to compatibility or performance issues in some cases.

[rafael2k]

@toncho11, I tested with apache and nginx, stock configuration, all good.
HTTP/1.0 should be support by virtually all servers. Nevertheless, adding HTTP/1.1 would not be difficult, as upstream libwww already has support to it (and it is not rocket science anyway).
: )

[toncho11]

Do you still have this problem with the second connection failing? It could be because one is expecting http 1.1 and the other 1.0.

[rafael2k]

Hi @toncho11, it has nothing to do with HTTP itself. The software is locking at bind() call here.

[toncho11]

Does it work now after: #2267 ?

[rafael2k]

Yes!

[rafael2k]

No more locks on ELKS.
: )

Now I need to fix the curses.

[toncho11]

How do you manage the lack of DNS? If you "click" on a link you will need to get the IP address of this new web-site - a new IP address. Hmm or maybe they are all https, so you do not click on them.

[ghaerr]

How do you manage the lack of DNS

What lack of DNS? I wrote a DNS resolver function in libc several years ago: in_resolv(). For details on how to use it, look at the nslookup network application in elkscmd/inet/nettools/nslookup.c.

[toncho11]

Nice! I did not know about it. So now we have everything for bobcat to work :)

[rafael2k]

How do you manage the lack of DNS

What lack of DNS? I wrote a DNS resolver function in libc several years ago: in_resolv(). For details on how to use it, look at the nslookup network application in elkscmd/inet/nettools/nslookup.c.

Oooops, I'm sorry about that, I haven't realized it. I'll wire up DNS support.

[rafael2k]

How do you manage the lack of DNS? If you "click" on a link you will need to get the IP address of this new web-site - a new IP address. Hmm or maybe they are all https, so you do not click on them.

If you "click" a relative link, it just works, like in www.frogfind.com. An absolute link using IP works, while using a name will work soon, after I implement name resolving using in_resolv().

[rafael2k]

I can confirm DNS is working fine with bobcat on ELKS! I made a kernel build removing some drivers I don't use, and now I can run bobcat without changing kernel parameters in /bootopts.

[toncho11]

Do you use copy.sh or qemu to test?

[rafael2k]

I'm using qemu to test. What is this copy.sh?

[rafael2k]

Basically, the engine is working great, just the curses needs a bit more, especially for forms and page navigation with directional keys.
After the interface is working fine, I think adding HTTP 1.1 and a subset of HTML 4 should be relatively straight-forward.

[toncho11]

Go here: https://copy.sh/v86/?profile=custom
Scroll down. You will find buttons for attaching both floppy and HDD ELKS images.
It works great. The virtual machine is actually executed in your browser.
Only problem so far is the lack of com mouse (on the ps2 protocol) which affects the use of Micro Windows.

[rafael2k]

You can also use the latest kernel build on 386 which has the hma=kernel option in /bootopts and this will give you another 64 KB!

Cool, this really helps.