Describe Dynamic Backend Interaction

[Savio-Sou]

Create spec on how dynamic backend loading will work with nargo

Think through different systems and produce system specific work items as needed

Blaine:
Rust dynamic linking looks really difficult. They have dylib and cdylib but dylib is extremely unstable and only should be used by the Rust compiler team (it will likely break between even the smallest Rust version changes). So the only real option is to generate a cdylib which has a stable ABI, but restricts the types to things C can handle.

There is a utility library (https://github.com/rodrimati1992/abi_stable_crates) for producing and working with "stable ABIs" but it very macro heavy and relies on specific layout knowledge. It also seems that you also have to wrap Enums into their NonExhaustive enum so variants can be added without disrupting their stable ABI.

Once all that is sorted, a library like dlopen or libloading could be used to dynamically load the backend.

Additional resources:

• http://blog.pnkfx.org/blog/2022/05/12/linking-rust-crates/
• https://michael-f-bryan.github.io/rust-ffi-guide/overview.html
• https://zicklag.github.io/rust-tutorials/rust-plugins.html

Alternatively, the backends could all be WebAssembly, but I don't think that's a viable solution because it becomes too slow for larger circuits.

Koby:

What if we would go IPC route instead?

Available options would be:

1. Simple Process Invocations w/ passing Command Line Params
2. Pipes - works locally only
3. TCP (ie. gRPC or ZeroMQ + ProtoBuf, better yet libp2p) which would allow for remote invocations, would support more commercially friendly patterns

The route depends on what Use Cases we see before Nargo. Is it just supporting development process? Or is it potentially useful in production environments?

[kobyhallx]

At the first attempt:

1. Nargo will invoke a backend as a subprocess
2. Bunch of environment variables are going to be set for child process, eg. where to store artifacts and what their name might be on fs.
3. A backend, an independent system level executable will process artifacts

[Savio-Sou]

Seems like a duplicate of noir-lang/noir#2164.

@kobyhallx would you like to copy whatever's still relevant in this Issue over there, then we can close this as obsolete?