Update: We're trying to finish this PR before July!

edit: well that didn't happen...

This is a massive PR already, but it still has a ways to go. I'm not exaggerating. Not ready for review yet, here be dragons.

Here are some interesting features added in this one:

• Big perf gains, 2-3x faster on some benchmarks, thanks to improving the way functions are called (among other things)
• Support for records, paving the way for a module system and type system.
• A number of optimizations, including less copying during lexing and parsing.
• A bytecode verification function, which will address Fully annotate unsafe code paths #44 as code will be verified before running.
• Better Wasm support thanks to less dependence on std.
• Support for token-based macros.
• Completely rewritten lexer for better unicode support, ~5x perf gains.
• Support for alt-radix integers, like 0xFF.
• Property-based testing for fuzzing throughout pipeline. (In progress)
• Move passerine and aspen and related crates (Qualm eventually) into a monorepo.
• Macros to convert Rust data to Passerine data and back again.

This PR still has a long way to go, but is getting closer to completion. Here's what's left:

Phase 1

• Refactored the representation of syntax trees in the compiler, to reduce boilerplate
• Rewrote lexer to be token-based macro friendly
• Refactored parser to work with new lexer
  • Refactor the way spans are represented for richer errors
  • Take advantage of new error format in parser
• Reintroduce rest of compilation pipeline, ensure it builds
  • Lex
  • Read
  • Parse
  • Desugar (type definition mismatch oversight, working on it now)
  • Hoist
  • Gen
• Write tests for new pipeline
  • Lex
  • Read
  • Parse (only unit tests so far, need proptests)
  • Desugar
  • Hoist
  • Gen
• Test pipeline E2E
• Get snippet tests all working
• Clean up and document new code
• Remove overly generalized compilation pipeline
• Clean up library exports
  • Figure out what really needs to be in passerine_common.
• System Injection
  • Figure out Rust side of things
  • Proof of concept (in private pnx repo)
  • Writeup (WIP)
  • Remove magic (finally!)
  • Conversion for builtin types
  • Conversion proc derive macro
  • Add effect (eventually to be replaced with type)
  • Switch existing system code over to effects
• Look at mlua/hlua, rhai, and pen to determine what the high-level API to using Passerine from Rust should look like.
  • Calling Rust from Passerine
  • Calling Passerine from Rust
  • Passing data to and from Passerine/Rust
  • Sandboxing side effects
  • Inspecting and modifying the results of the top level (requires modules)
  • Binding handlers through system injection (implemented in pnx, needs to be transferred)

Phase 2

• Introduce a token-based macro system
  • In between lexing and parsing
  • That is hygenic (TODO: how to define hygine)
• Introduce prelude
  • standard library and core
  • reintroduce syntax-based macros in terms of token-based macros

Phase 3

• Finish implementing type ... and record
• Introduce rough modules
• Match expressions using macros and fibers
• Polish implementation write some projects to test for bugs
• Polish standard library and prelude a tad

Phase 4 (Maintenance stuff, post merge)

• Polish high-level API, write documentation and examples
• Go over README and correct anything that is inaccurate
• Move README Overview to the codex and link to it
• Update the code example in the repo and on the website
• Redo the website, add a 'try it' wasm-based playground

These phases are loosely ordered. Upon completion, this PR will be the 0.10.0 release. As we near phase 2 or phase 3, if you'd like to test the changes or help add some standard library functionality, please let me know so I can set up some time to walk you through the code base.

Thanks, have a nice day!

cc: @ShawSumma @Plecra @IFcoltransG @spaceface777 @hhhapz