Investigate if we can use Rust parallelism from a Ruby API

[vinistock]

One of Rust's great advantage is safe parallelism/concurrency. If we build a native extension using Rust, are we able to use parallelism from a Ruby API? Or will the GIL (or something else) somehow prevent that from happening?

I assume that, as long as we are only interacting with Rust's data structures and not crossing the boundary to the Ruby VM, it should be possible.

Investigate and provide the answer as to whether we can architect our Ruby APIs using Rust's parallelism.

[jesse-shopify]

It may be worth considering 2 types of parallelism in Rust:

• without runtime (aka rayon) - I think this is the better place to start
• with runtime (aka Tokio) - downside will be the dependence on the runtime and having to spin up the runtime. Upside could be access to async libraries.

[ChallaHalla]

Based on some experimentation, it seems that not crossing the Ruby VM boundary allows for true parallelism. Will confirm though before closing this ticket.

[ChallaHalla]

@jesse-shopify What advantages or tradeoffs are entailed with using rayon VS using std::thread?

[ChallaHalla]

Closing since we've confirmed that Rust parallelism can be used when invoked from a ruby API

[ChallaHalla]

Here are some examples of what Core activity looks like in HTOP for parallelism vs concurrency.

Parallel execution of threads in Rust:

benchmark_threads_rust.mov

Concurrent execution of threads (no parallelism) in ruby:

benchmark_threads_ruby.mov