Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Collapse OS in Forth #4

Closed
mistergibson opened this issue Oct 8, 2019 · 131 comments
Closed

Collapse OS in Forth #4

mistergibson opened this issue Oct 8, 2019 · 131 comments

Comments

@mistergibson
Copy link

@mistergibson mistergibson commented Oct 8, 2019

Please see this: http://www.camelforth.com/page.php?5

Forth is one of my past loves. It can be used for most things rather well and is the precursor to PostScript found in printers. Using Forth one could conceivably construct a Display PostScript like system to build a rudimentary GUI system, and provide print support.

Any chance of including a version of CamelForth with CollapseOS? I'd like to hear your thoughts.

Cheers, and great work on this project!

G.

@nastepnym
Copy link

@nastepnym nastepnym commented Oct 8, 2019

and mruby ;)
400KiB are needed

1 similar comment
@nastepnym
Copy link

@nastepnym nastepnym commented Oct 8, 2019

and mruby ;)
400KiB are needed

@Skrylar
Copy link

@Skrylar Skrylar commented Oct 8, 2019

I would suggest changing the question to one about including a FORTH at all.

FORTH having a low burden to get an environment running on exotic hardware is one of its most significant benefits.

Though mostly in that FORTH code is just barely a veneer over stack machine assembly. It might also be worth looking at old Z80 interpreters (should be able to find them around old infocom fandoms, since they like to keep that stuff alive) or various small VMs (RetroFORTH's Nga and Ngaro are well documented and tiny, Lua's VM design is very battle tested, and I think Janet's is basically Lua but on a stack machine)

You just need a VM which is easy to port, and then you can run whatever compilers on top of that 😉 Amusingly, the need to be portable across multiple different computing platforms (even though they were still mostly Z80's) is one of the reasons scripting used to be so prevalent in the Amstraad days.

@agsb
Copy link

@agsb agsb commented Oct 9, 2019

I agree, a minimal stack forth can support a environment OS and be used to make compilers and "primal" archive functions.

@hsoft
Copy link
Owner

@hsoft hsoft commented Oct 9, 2019

FORTH came a lot in the different comments I got. I knew it by name, but I've been convinced to look more into it, which I started to. We'll probably want a FORTH interpreter, yes.

@paulidale
Copy link

@paulidale paulidale commented Oct 9, 2019

Forth is often denser than assembly code. It's very portable and only takes a tiny kernel. It seems absolutely ideal for this project. For example: the book Threaded Interpretive Languages covers most of the implementation details.

The big problem with Forth is that it can often be difficult to understand. Easer to understand but not quite so compact and somewhat slower is LISP. It would be another good language for this kind of project -- again it only requires a tiny kernel.

@Skrylar
Copy link

@Skrylar Skrylar commented Oct 10, 2019

Lisp has more compiler overhead, kind of. The dictionary would be bigger (because they use full names like read-file instead of something like F<) but compiled code would ultimately be pointers in both. And having to read an entire function definition before compiling it (I suspect Forth can just 1:1 words with machine code, which does work even if the result is sub-optimal.)

Not much of a reason it should run slower afterwards though. You could probably compile the Lisp to Forth in the background.

@lmamakos
Copy link

@lmamakos lmamakos commented Oct 10, 2019

Including a variation on Jones FORTH targeted to the Z80 would be a nice addition.

See https://github.com/nornagon/jonesforth/blob/master/jonesforth.S for most of the machine-specific code and what I think is the best example of literate programming I've ever seen! You start reading the comments at the top, with more and more bits of code interspersed and then by the time you reach the end of the file, you have the core of the FORTH kernel.

The mode of presentation makes a great tool for someone that then would want to target some other hardware architecture.

@lmamakos
Copy link

@lmamakos lmamakos commented Oct 10, 2019

Any chance of including a version of CamelForth with CollapseOS? I'd like to hear your thoughts.

The other nice resource that accompany's CamelForth is a series of wonderful articles written about its design and implementation choices. Starting at https://www.bradrodriguez.com/papers/moving1.htm and following parts. Again, another great thing to have at your disposal when considering porting the FORTH kernel to a different CPU architecture.

@jes
Copy link
Contributor

@jes jes commented Oct 16, 2019

We'll probably want a FORTH interpreter, yes.

How does this help preserve humanity's ability to program microcontrollers with low-tech tools?

I'm not saying there shouldn't be a FORTH interpreter! Just perhaps that the goal of the project isn't as clear to many of us as it could be.

@hsoft
Copy link
Owner

@hsoft hsoft commented Oct 16, 2019

@jes from what I understand (still exploring this field), bootstrapped FORTH could be as compact if not more compact than straight ASM. This makes it an interesting path to explore.

@agsb
Copy link

@agsb agsb commented Oct 16, 2019

@jodavaho
Copy link

@jodavaho jodavaho commented Oct 17, 2019

Every programmer (or very nearly) learns some assembly. It's ugly and painful but easy enough to grok once you realize how dumb a computer is (that's the point of learning assembly, I think). A quick poll around (large company of programmers) reveals nobody knows any forth. Perhaps it is mainstream in some community, and perhaps it is the best tool for this job, but if accessibility is the goal, then this goes against the utility to any non-specialized technologist / programmer post-collapse.

This is not an argument against FORTH, which might be the shiniest bit of bits around, but an argument against the accessibility penalty of collapseos. Granted, I'm a non-contributing spectator at the moment, so take this with a grain of salt.

@hsoft
Copy link
Owner

@hsoft hsoft commented Oct 19, 2019

Thanks @lmamakos for the Moving Forth link. It's very interesting.

I've been exploring Forth and it's a fascinating field, but too time consuming to explore and I'll think I'll stop there. So far, what I've read on that subject made me come to a few conclusions. Please, someone more knowledgable correct me if I'm wrong.

Forth only deals with 16-bit numbers

Even on 8-bit CPUs, it doesn't seem imaginable to have a forth with 8-bit cells. Making all operations of a 8-bit CPU 16-bit seems a bit wasteful to me. Doesn't Forth incur a heavy performance penalty compared to hand-written assembly?

Forth has many layers of indirection

I looked at how threads work in Forth and although we can't call this an interpreted language, it seems to me that there's much more indirection than in assembly language we see in current Collapse OS code. That would, again, incur a heavy performance penalty.

Is Camel Forth self-hosting?

Unless I'm missing something, CamelFoth is written in z80 assembly, but it doesn't include a z80 assembler. To make CamelFoth self-hosting, a z80 assembler would need to be written in Forth.

Conclusion

Unless I'm missing something (I probably am, but please correct me), these points above would definitely rule out Forth as being the "foundation language" for Collapse OS: too wasteful with regards to resources.

If someone wanted to port CamelForth to Collapse OS' (which would probably be a minor task considering that it can be trivially made self standalone), I'd say it would be a nice addition, but as @jes says, I'm not sure how it brings Collapse OS' closer to its goal.

Side note: when trying to answer "does someone know of a z80 assembler in Forth?" question, I came across https://github.com/siraben/zkeme80 which looks nice. But it seems that its assembler is written in Scheme, so the question still stands.

@hsoft
Copy link
Owner

@hsoft hsoft commented Oct 19, 2019

Some further research I've made led me to https://www.bradrodriguez.com/papers/tcjassem.txt (Brad again!) which would seem to indicate that Forth is well suited for writing assemblers.

@hsoft hsoft mentioned this issue Oct 31, 2019
@Allisontheolder
Copy link

@Allisontheolder Allisontheolder commented Nov 12, 2019

I've seen stand alone forth that fits in something like 8k rom.
The 1802 based ELF2000 system has Forth in eprom, along with a
monitor, bios, library routines, and even tiny basic and an editor.
Forth is a very compact language.

I'm not a forth fan it makes my head hurt but have used it and
it does seem efficient. Much more so than NASCOM basic.

Allison

@agsb
Copy link

@agsb agsb commented Nov 12, 2019

@Allisontheolder
Copy link

@Allisontheolder Allisontheolder commented Nov 12, 2019

A lot of it will not fit stuff seems to come from people only used to PC (winders/linux/Mac)
space where things are huge and bloated. The micro world (6502/1802/8080/z80/6800)
people were used to limited ram and efficient code was the rule so much of the stuff from
then will fit easily. Forth was one such item of code.

Allison

@Allisontheolder
Copy link

@Allisontheolder Allisontheolder commented Nov 16, 2019

Looked in my Forth folder and:
Camel Forth Z80 for CP/M, NS*dos and a standalone version
with a bag and tag file system for a simple 765 based FDC.
All three came form the Teserect BBS around '84ish.
The largest is the standalone and its under 8K.
They were PD then.

There was an 8080, PDP-11(RT-11 ) and 6502 (apple prodos) version as well.

@PeterForth
Copy link

@PeterForth PeterForth commented Dec 10, 2019

Openboot Openbios and OpenBsd are examples of using Forth as OS. Camelforth will start a Forth from hex code, as also any FIG forth for Z80, there are many free listings. Have a look at my collection of videos on Youtube so you can learn more about forth. This is my contact form you can reach me by email and I will answer your questions and try to help. https://sites.google.com/view/win32forth/home/contact (Forth was 16 bit till 90´s then when the 32 bit CPUs arrived, Forth became the facto 32 bit as standart. There are also 64bits Forth, and no reason to make 128 bits forths or higher, since forth is very flexible and real easy to implement on any hardware) Another good example of the Z80 Forth systems is the Jupiter Ace https://en.wikipedia.org/wiki/Jupiter_Ace

@gitjeff2
Copy link

@gitjeff2 gitjeff2 commented Jan 23, 2020

I just want to bring up the fact that lbForth Lite exists. It's a Forth cross-compiler for tiny devices. The cross-compiler targets many of the CPU's @hsoft appears to be interested in getting CollapseOS up and running on. It looks like adding additional CPU targets such as the Z80 wouldn't be all that difficult for someone who has a decent grasp of that architecture.

From what I can see the current list of CPU's is:

  • MOS 6502
  • C8051
  • AVR
  • MSP430
  • PDP8
  • PIC
  • STM8
  • THUMB

While this won't directly help you with a native Forth compiler, getting cross-compiler going is likely to be a huge help every time you add a new architecture. To help you wrap your head around Forth there is an excellent 10 article series that walks the reader through writing a Forth compiler. This is basically the "guts" of CamelForth (GPL3) which someone else already mentioned. Here is the link to the CamelForth source code.

@hsoft
Copy link
Owner

@hsoft hsoft commented Mar 4, 2020

I'd like to let people know that I started, as an experiment, a reboot of Collapse OS as a Forth: https://github.com/hsoft/collapseforth . The goal is too bootstrap this Collapse OS-to-be entirely through that evolving Forth.

I'm learning Forth as I go. I'd be curious to hear about what people experienced in Forth think of this approach.

@hsoft
Copy link
Owner

@hsoft hsoft commented Mar 8, 2020

The approach I took with that "collapseforth" reboot was a dead end, but it made me learn a bit more about Forth and I like it. I've added apps/forth, a new z80 implementation of Forth, in Collapse OS directly. With time, I can very well see it take a central place and make Collapse OS self-hosting in an even compacter way.

@hsoft hsoft mentioned this issue Mar 12, 2020
@hsoft hsoft changed the title Question : any chance of including CamelForth? Collapse OS in Forth Mar 12, 2020
@hsoft
Copy link
Owner

@hsoft hsoft commented Mar 12, 2020

I'm changing the title of this issue because I can see the distinct possibility of making Forth take over Collapse OS. My expectation is that, because higher order words are more powerful, they are also more compact.

With any other higher order language, this high level comes at a prohibitive cost in terms of implementation complexity (and thus RAM usage), but not with Forth, which is what makes it so interesting.

Collapse OS currently self-hosts with about 10K of ROM and 8K of RAM. The vast majority of that space is zasm's complexity. I believe that with Forth very quickly (in terms of complexity) bootstrapping into higher order words, it makes its "self complexity" (the complexity it needs to self-host entirely) much lower, I can achieve similar or lower requirements and get a high-level language (and arch-independent!) as a bonus. This makes Forth very exciting.

I'm not certain of success yet, but I'm optimistic and excited. Expect Forth to "eat away" (I will not do a reboot, just an incremental "code cancer") current ASM code.in the following days.

@hsoft
Copy link
Owner

@hsoft hsoft commented Mar 12, 2020

By the way, sorry to Forth enthusiasts for not having seen its potential early. It's difficult to see it without diving into it. It's such a wicked way to approach computing in general. I'll have to supply Collapse OS doc with good Forth introductory material.

@agsb
Copy link

@agsb agsb commented Mar 12, 2020

@hsoft
Copy link
Owner

@hsoft hsoft commented Mar 12, 2020

@agsb yes, it's been on my reading list. I've started reading it, but my hands are too dirty playing with code. If I were reasonable, I'd take the time to step back and read this...

@piranna
Copy link

@piranna piranna commented Apr 27, 2020

Awesome! :-D

@ghost
Copy link

@ghost ghost commented Apr 27, 2020

It doesn't seem very compact.

I thought it was interesting that a (very light) lisp-like or imperative language might by able to be evolved out of the work you're doing creating forth, possibly for scripting within a minimal shell environment.

@ghost
Copy link

@ghost ghost commented Apr 27, 2020

Corresponding HN discussion : https://news.ycombinator.com/item?id=22935650

@gitjeff2
Copy link

@gitjeff2 gitjeff2 commented Apr 28, 2020

I'm not sure how much this applies to the Z80, which you're targeting first for entirely understandable reasons. But, there is a very interesting article that will help increase the performance of the 6502 port when it gets here: Zero-Overhead Forth Interrupt Service on 65C02.

I'm not sure if it applies in anyway to the Z80.

@hsoft
Copy link
Owner

@hsoft hsoft commented Apr 28, 2020

It looks like the concept can be applied to z80 as well. It is also very relevant to Collapse OS' needs: I wrote the ACIA interrupt routine in assembly precisely for the reasons cited.

However, calling it zero-overhead is a bit of a misnomer: the NEXT routine has to poll for interrupt requests. The question is whether that polling is worth it.

Right now, only the ACIA drivers use interrupt requests, I'll wait until there's a few more of them before asking myself the question seriously. But thanks for the reference, it's certainly a good one.

@f4grx
Copy link

@f4grx f4grx commented Apr 29, 2020

Also usable in my own implementation on HC11 and soon-to-be on msp430, which is now able to find words!

Thank you for the awesome IRQ management link.

So basically, native IRQ routines raise a flag and the inner interpreter just EXECUTE a specific forth word if the flag is set. The next word will naturally get executed on the next call to NEXT without even having to save/restore IP! This is a close as a native implementation as it could be, I love it!

@gitjeff2
Copy link

@gitjeff2 gitjeff2 commented Apr 29, 2020

Thank you for the awesome IRQ management link.

Sure @f4grx! I don't have the cycles to submit code just now, which sucks because I have a few ideas I'd really like to implement. But, I can send useful resources and do other things to stay active and help the project.

@gitjeff2
Copy link

@gitjeff2 gitjeff2 commented May 1, 2020

Given the anticipated hardware ecosystem of CollapseOS, something like 6502 binary relocation format might be useful. Much like the so-called "Zero-Overhead Forth Interrupt Service" article, this article is 6502 specific. I hope it's useful in helping to reduce assumptions about a given system's memory map regardless of architecture and increase code portability.

Memory map assumptions plagued CP/M again and again. Wildly incompatible disk formats were an issue too, but it sounds like you're focusing on FAT or ex-FAT for now. So, that's not an issue.

@gitjeff2
Copy link

@gitjeff2 gitjeff2 commented May 1, 2020

The RTX2010, an 8-bit processor that supports direct execution of Forth code. Lots of technical details are readily documented. Effectively, Forth is its machine code. I wonder if this can be recreated efficiently in purely TTL. Apparently, the RTX2010 chip has been used by NASA because it can be easily hardened.

@PeterForth
Copy link

@PeterForth PeterForth commented May 2, 2020

@Picatout
Copy link

@Picatout Picatout commented May 2, 2020

@PeterForth, Except for GA144 do you know any CPU or MCU with a native FORTH core still available as 'active part' today? (No FPGA).

@PeterForth
Copy link

@PeterForth PeterForth commented May 2, 2020

@piranna
Copy link

@piranna piranna commented May 2, 2020

Propeller or Pharlap (propforth and others) still delivers a modern Forth native CPU, but not Forth architecture CPU. en.wikipedia.org/wiki/Parallax_Propeller It is really an impressive chip for automation and controll, is also very fast even not = to a Forth-architecture cpu. That CPU is also free available as FPGA Verilog core.

Propeller Parallax is available as a Verilog core?!??! Where?!?!?!?! 😲 What license does it have?

@piranna
Copy link

@piranna piranna commented May 2, 2020

On August 6, 2014, Parallax Inc., released all of the Propeller 1 P8X32A hardware and tools as open-source hardware and software under the GNU General Public License (GPL) 3.0. This included the Verilog code, top-level hardware description language (HDL) files, Spin interpreter, PropellerIDE and SimpleIDE programming tools and compilers.[3]

OMG, that Verilog files would be really nice to read, I remember to read that the cogs where created using basic gates instead of high level code, making the DIE layout smaller and faster :-D

@Picatout
Copy link

@Picatout Picatout commented May 2, 2020

@PeterForth ,
Thanks for information and links.

@hsoft
Copy link
Owner

@hsoft hsoft commented May 3, 2020

Things are getting deliciously serious on the TRS-80+Forth side. With the new indirections I've added to KEY and EMIT, I can easily take control of the prompt from RS-232, and with the new blkup tool I've just pushed, I can send the blkfs directly to floppy. It works, I now have a floppy with the first 10 blocks of Collapse OS fully working on the TRS-80!

With a completed editor and some smart management on the blocks side (to avoid having to swap disks all the time), with the addition of some kind of daemon on the "modern side" listening to the serial com for incoming blocks to spit to blk/, I could easily see myself in a setup where I work on Collapse OS from within Collapse OS on a TRS-80 4P. sexy...

@Picatout
Copy link

@Picatout Picatout commented May 5, 2020

hsoft, "Things are getting deliciously serious on the TRS-80+Forth side. "
Do own the hardware or is it an emulator?

@hsoft
Copy link
Owner

@hsoft hsoft commented May 5, 2020

On the real thing of course. I'd have a hard time qualifying running Collapse OS on an emulated TRS-80 4P as "delicious" :)

Yup, Collapse OS now spans on 5 1/4 floppies and my last commit, the one about ACC initialization, was first fixed on the machine itself (I don't notice these initialization issues on emulator because memory is zeroed in there, something that doesn't happen on the real thing) using the in-system editor to write it to floppy and then re-LOAD it t confirm the fix... I'm getting there.

@hsoft
Copy link
Owner

@hsoft hsoft commented May 10, 2020

TI-84 recipe is now working on the Forth branch. There's not much left to implement before Forth is on par with master and can thus replace it: SMS recipe and AVR assembler.

@hsoft
Copy link
Owner

@hsoft hsoft commented May 11, 2020

@TalosThoren your video at https://twitter.com/talosthoren/status/1259825070509297664 looks very good! thanks for making it. I haven't finished watching it, but I presume that you end up being able to boot Collapse OS :) I'll link it in the project's README. It's so far one of the best introductory material I've seen so far.

EDIT: oh, I thought you got to Collapse OS prompt. You still have this whole write-to-EEPROM challenge to meet. Let me know if you find the documentation lacking. I'm interested in knowing how newcomers can manage following the recipe.

@hsoft
Copy link
Owner

@hsoft hsoft commented May 14, 2020

Collapse OS now cross-compiles in a single stage! On april 12, I was commenting that I managed to bootstrap Collapse OS using a hybrid xcomp+relink approach, which was the approach I took until now. Cross compiling the whole base set of words was too hairy.

However, now that things are stabilizing, I took a second look at the situation and with careful threading, I could manage to organize core words in a way that makes them 100% cross compilable.

This makes the bootstrapping process much leaner.

@f4grx
Copy link

@f4grx f4grx commented May 14, 2020

That's great!
What about text editing?

On my part I have a working compiler, which is also great for me.

@hsoft
Copy link
Owner

@hsoft hsoft commented May 14, 2020

The editor is partly implemented, following Starting Forth's description. Inconvenient but usable.

@hsoft
Copy link
Owner

@hsoft hsoft commented May 16, 2020

Forth Collapse OS runs on my Genesis with a gamepad!

This brings me really close to parity with the master branch. The PS/2 sub-recipes from SMS and RC2014, the AVR assembler, and I'm done!

@piranna
Copy link

@piranna piranna commented May 16, 2020

Forth Collapse OS runs on my Genesis with a gamepad!

How are you using it? An on screen keyboard or something similar? :-)

@hsoft
Copy link
Owner

@hsoft hsoft commented May 16, 2020

The recipe describes how it works.

@hsoft
Copy link
Owner

@hsoft hsoft commented May 21, 2020

I've completed the conversion of the PS/2 interface recipes in RC2014 and SMS. This completes the conversion to Forth! Now I only need to clean up things a bit, beef up the usage docs section, and switch the master branch.

@hsoft
Copy link
Owner

@hsoft hsoft commented May 22, 2020

Forth takeover is complete! closing.

@hsoft hsoft closed this as completed May 22, 2020
@f4grx
Copy link

@f4grx f4grx commented May 22, 2020

Congrats and thank you for making me dive in forth. I dont regret it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests