Bindings IR

[bendk]

Adding a new system for bindings generation, modeled after a compiler pipeline with a lot of small steps. See the internal docs in this PR for a high-level overview.

Planning out the work

I have a patch in uniffi-bindgen-gecko-js that builds on this to rework those bindings. I'm hoping to move Python/Kotlin/Swift over to using this system, but there's no code to do that yet. I'm pretty confident that will work though, because it's working with JS and the previous version of this was working with Python.

This means that this PR only contains the general half of a pipeline, to see how this could be finished for a particular bindings language, check out the patch for the gecko js bindings. I recommend focusing on the code in the pipelines directory and the JS templates, the C++ code is weird and specific to the Firefox codebase.

The way I'd like to land this all is:

• Merge this PR into UniFFI
• Make a UniFFI 0.29.2 release with the code
• Start using this code in Firefox
• Work to move the other languages over. I'm planning on targeting 0.30.0 with that work, but I don't think there's any particular urgency to it.

Comparison to previous works

This is a continuation of #2333 and several other attempts that I never ended up pushing out.

The IRs

The IR types are mostly similar to before, with one big difference: IRs contain items from every crate involved, Module nodes are children of the Root node. My feeling is that this will set us up to finally tackle issues that require cross-module coordination like #1896 and #2430.

The passes

I tried sooooo many ways to express the pipeline logic in a nice way. In the end I was inspired by the nanopass framework, where you have a bunch of little passes. That keeps the logic modular and readable. The modules in uniffi_bindgen/src/pipeline/general, are only doing a single thing. Before, we had one big pass that converted everything, which was harder to understand.

The downside of a nanopass framework is that it's pretty easy to make rustc explode. My initial try created a separate set of IR node type for each pass, but once I got to a dozen or so the build time for uniffi_bindgen went to about 30s. Then it started getting killed by the OOM killer and I decided it was time for a new direction.

To avoid this, the new system creates 2 IRs, but then splits up the work into multiple pass functions. Even though each step works on the same IR, it still kind of feels like they're transforming the types. Instead of adding a new field, they now set a value for a field that was previously empty.

This achieves more-or-less the same effect as the nanopass system, but with much faster compile times. My not-very-scientific measurement is that it went from 12s before to 17s now. That's not great, but if this is successful we can hopefully delete a bunch of the old code and get times down to something reasonable again.

The CLI

I replaced the peek and diff commands with the pipeline command. If you have time, read the internals doc about it and take it for a test drive. IMO it's pretty fun walking through each pipeline step.

[bendk]

Updating this PR to use less macros and syn parsing. Now, instead of wrapping everything with the ir! and using ir_pass! to generate the pass IR, there's just derive macros. One implements the node trait and one generates the macros to construct the nodes. This means instead of having ir_pass! auto-generate everything, you need to write out the pass IR types. These changes were based on my experience with JS, which was mostly positive but showed some of the issues with the last design.

The main issue was that when we were auto-generating the pass IR types, we would copy all the #[derive] attributes from the output IR. This caused issues since I wanted to #[derive(rinja::Template)] for the Gecko-JS IR types, but it doesn't make sense to for the pass IR types and in fact errors out because there's no filters module.

The new system requires extra boilerplate compared to the last, but it also feels much less magical. I like this trade-off overall, even when you ignore the issue with rinja::Template.

There's no longer specialized code to convert uniffi_meta types into the initial IR. We just derive Node for those types and then we can automatically do the conversion. This means that the foundational IR pipeline code needed to move out of uniffi_bindgen, since uniffi_meta can't depend on uniffi_bindgen without creating a circular dependency. I created the uniffi_pipeline crate to hold that code.

The other improvement is that the new Node types can operate more dynamically, i.e. the new pipeline is based around Box<dyn Node> rather than any concrete type. I think this makes the pipeline code nicer.

[mhammond]

This is really quite remarkable and I know is the result of a lot of work - thanks!

While it's going to be an interesting (but optional!) ride for existing bindings, I'm 100% sure this is going to improve making and rationalizing about bindings improvements.

[mhammond]

bindings using this wouldn't face #1789 right?

[bendk]

My goal was to try to duplicate the current logic with this one, so I think that would still be a bug but I'm not totally sure.

[badboy]

In yesterday's UniFFI meeting we discussed this.
We agreed that it's worth experimenting with this further and for that to merge this PR.
The mozilla-central code can make use of it to evaluate it further. We also will try to move over one of the built-in generates (most likely Python). This should give us an idea of (a) whether it's possible, (b) it's a real improvement for UniFFI, (c) gauge the effort this takes.
This is not yet a final commitment on this new IR approach. The recommendation for any external users is to NOT use it yet.

@bendk: I hope this summary is accurate. Correct me if there's something wrong.

[bendk]

Push one more change to this:

• Updated the docs to make it clear this is experimental and not recommended for outside contributors yet. The docs are purposely not linked in the TOC and there's no CHANEGLOG entry for this reason.
• Lean on Default::default rather than defining the node construct macros (TypeNode { ty, ..TypeNode::default } rather than TypeNode! { ty }. This is one less macro to understand and maintain. In some places it's a bit more verbose, but not by that much.
• Pass the uniffi.toml contents down through the pipeline so the language-specific pipelines have easy access to it.

My plan is to merge this tomorrow unless there are any more suggestions/concerns.

[bendk]

Push a couple more tweaks:

• Clippy fixes
• Use RustFfiFunctionName, FfiStructName, and FfiFunctionTypeName in all places where we have those names -- for example, FfiFunction::name. This makes it easier to write passes that transform those names.

I'm still hoping to merge this one tomorrow.

[bendk]

I was out on Thursday, but now I'm back and can confirm that the Gecko JS bindings work with this new version. Let's merge!