Thoughts on maintaining the `SpirvBuilder` interface

[tombh]

I've been trying to get a rust-gpu shader crate compiled with cargo gpu, but the shader still relies on an oldish rust-gpu that uses the nightly-2023-09-30 toolchain. I've tried compiling with a newer version of rust-gpu (namely the current default in cargo-gpu) but the shader seems to generate a new compile error. I'll submit an issue about that later on rust-gpu itself.

I was hopeful that cargo gpu's --spirv-builder and --rust-toolchain arguments would help here, but of course I soon ran into the fact that older versions of spirv-builder don't have the same method interfaces. For example, SpirvBuilder::new().rustc_codegen_spirv_location() didn't exist. And so even though cargo gpu did everything else right, checking out the old rust-gpu version, installing the old toolcahin (that is so awesome BTW!), it couldn't actually do the shader compilation simply because SpirvBuilder had changed.

I suspect this might be something we could run into more in the future. Sure we can just freeze the interface, but that's not realistically sustainable in the long run. So what other options are there? One I can think of is putting spirv-builder into its own repo, or maybe even this repo? That way spirv-builder and rust-gpu can be downloaded separately. But then maybe that just moves the same problem to the spirv-builder-rust-gpu interface?

PS: I wonder if a better name for --spirv-builder could be --spirv-builder-source or --spirv-builder-version?

[schell]

The root at the center of this tangled mess you're unraveling is that all the crates in the rust-gpu repo are tied to a specific nightly toolchain. rust-gpu crates have to move in lockstep with regard to this toolchain. There's no good way around that and it's one of the reasons I was so eager to pick up development of cargo-gpu - I really don't want my entire project pinned by an old(ish) nightly!

You have a great point that the interface to SpirvBuilder is a problem. I don't think moving the crate out of rust-gpu would help much simply because it still needs to move in lockstep with the other crates in that repo.

But I think we can get around the issue by doing some patches. We could patch the previous crates.io releases to include the new parts of the interface and the updates that allow the path of the rustc_codegen_spirv dynamic library to be passed explicitly. This would allow previous versions of spirv-builder to be compiled by cargo-gpu.

But even before that - we might be able to have cargo-gpu set some environment variables or create a symlink that might enable old versions of spirv-builder to compile. I'll do some investigation.

Thank you @tombh !

[tombh]

Glad I found something useful!

I'm very much of agreement that keeping all the nightly toolchain requirements out of shaders is a good idea.

Oh yes, I hadn't thought of crate patches. Though in my case I'm pinning to a git revision so it's not 100% certain that a released version would exactly have everything I need. Maybe only 99% ha.

Another idea, what if rust-gpu was checked out twice, once for the rustc_codegen_spirv required by the shader and once again for the spirv-builder required by spirv-builder-cli? They could both be compiled with their own toolchain. But I guess that's breaking out of the lockstep you mentioned.

I did actually manage to at least override .rustc_codegen_spirv_location() with:

const fn dylib_path_envvar() -> &'static str {
    if cfg!(windows) {
        "PATH"
    } else if cfg!(target_os = "macos") {
        "DYLD_FALLBACK_LIBRARY_PATH"
    } else {
        "LD_LIBRARY_PATH"
    }
}

But the compiler wasn't complaining about everything at once; .target_spec(), .shader_crate_default_features() and .shader_crate_features() are also new and don't seem to be able to be overriden with ENV vars.

[tombh]

Oh and another idea. What if we added a CLI interface to the current spirv-builder that would be used for all current and future spirv-builder-cli versions? That way the "RPC" becomes just CLIs, which also isn't 100% sustainable, but is significantly more forgiving than Rust, whilst still being maintainable with some creativity. And then previous versions of spirv-builder, that don't have the CLI interface, are called via the current Rust SpirvBuilder interface.

[schell]

Just for clarity @tombh are you suggesting we create a crate to hold the cli interface to spirv-builder-cli and then have both cargo-gpu and spirv-builder depend on that?

[tombh]

I think I figured it out by just using Cargo features! #20