Initial Rust project setup and POC implementation of the Forthic toke…

[joshvfleming]

…nizer

Testing

For now I've left the test-rs target out of test-all becuase it requires setting up a Rust build environment, and I don't want to break the current build. It can be run individually, though:

$ make test-rs

Forthic Rust tests
============
cargo test --manifest-path tests/tests_rs/Cargo.toml
   Compiling forthic-rs v0.1.0 (/Users/josh/Projects/forthic/forthic-rs)
   Compiling tests_rs v0.1.0 (/Users/josh/Projects/forthic/tests/tests_rs)
    Finished test [unoptimized + debuginfo] target(s) in 0.79s
     Running unittests src/lib.rs (tests/tests_rs/target/debug/deps/tests_rs-066189a2b06860a9)

running 6 tests
test test_tokenizer::tests::test_arrays ... ok
test test_tokenizer::tests::test_start_module ... ok
test test_tokenizer::tests::test_end_module ... ok
test test_tokenizer::tests::test_end_definition ... ok
test test_tokenizer::tests::test_basic ... ok
test test_tokenizer::tests::test_strings ... ok

test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Performance

I ran a basic benchmarking pass against the example Forthic programs. Times were averaged over 200 runs, and only the core loop was measured, so there's no interference from system startup time. Here are the results:

Program	Python (ms)	Rust (ms)
cache-example	0.0874	0.0131
gsheet-example	0.1230	0.0230
intake-branched	0.9363	0.0946
intake-multistep	0.7884	0.0864
intake-simple	0.7779	0.0839
jira-example	0.1770	0.0289
time-in-state	0.3440	0.0390
shakespeare [1]	545.3	60.16

• [1] A 1mb file containing text from Shakespeare (link).

Notes

• This is just the project setup and Tokenizer for now. The interpreter implementation is coming in a follow-on PR.

[rjose]

Looks great -- it's cool to see a new version of Forthic forming 👍🏼

It's interesting to see how cleanly the Python and JavaScript versions map into Rust. I'll have to study this more to understand the language. Maybe next time we chat, we can go through one of the files and I can ask you about the mental model of Rust and what some of the conventions are.

[joshvfleming]

Looks great -- it's cool to see a new version of Forthic forming 👍🏼

It's interesting to see how cleanly the Python and JavaScript versions map into Rust. I'll have to study this more to understand the language. Maybe next time we chat, we can go through one of the files and I can ask you about the mental model of Rust and what some of the conventions are.

Yep, the Tokenizer was more or less a direct translation from Python. It's going to be a different story with the Interpreter though, because Rust takes something of a stance against traditional OO patterns.

Absolutely, I think there's a lot of good stuff here to talk about.

BTW -- I updated the description with a quick benchmarking comparison.

[rjose]

Yes, I ran into a lot of pain with the Haskell version of Forthic. I was doing lots of dynamic things, which it was trying to prevent. Let me see if I can dig it up -- maybe some of the ideas there will be helpful.

Thanks for posting the benchmarks. It's neat to see how fast Rust is. (I think the intake-multistep cells need to be swapped)

[joshvfleming]

Yes, I ran into a lot of pain with the Haskell version of Forthic. I was doing lots of dynamic things, which it was trying to prevent. Let me see if I can dig it up -- maybe some of the ideas there will be helpful.

Thanks for posting the benchmarks. It's neat to see how fast Rust is. (I think the intake-multistep cells need to be swapped)

Whoops, you're absolutely right. Fixed.