How should dynamic linking work in Rust?

[jsgf]

If I set crate-type=["dylib"] in Cargo.toml, I would expect it to not only compile the top-level crate as dynamic, but also all its transitive dependencies, and link with them dynamically (ie, with -C prefer-dynamic semantics).

Right now if you try this, it builds all the dependencies as static, and statically links them into the dynamic crate it builds. If this dynamic crate is subsequently used to build an executable with a diamond dependency graph (ie two of its dependencies have the same transitive dependency), rustc fails because of the crate that's duplicated by being linked into two other dynamic crates. This happens most often with libstd, but it can happen with any shared dependency. This suggests that the default for any dynamic crate should be "prefer dynamic" for its dependencies, since the current default is only useful in very niche cases.

If you specify -C prefer-dynamic as a build flag, then it will not include the dependent crates in the dynamic output - but it also won't build the dependencies at all.

Further, if you specify crate-type=["dylib", "rlib"], I would like it to build the crate twice as static and dynamic (this does work currently), and all its dependencies (this does not).

Related: #3408

[alexcrichton]

Note that the current behavior exists for backwards compatibility at this point. Long ago a cdylib didn't exist so the only way to create a dynamic library was through the dylib crate type, so Cargo must preserve the current behavior today. That diamond dependencies don't work is a bug (rust-lang/rust#34909) and it should be fixable at any time really.

The use case for dynamic libraries in Rust has basically never been gamed out. As a result a bug like this is likely much broader in terms of "how should dynamic linking work in Rust" at all. I don't think that just building a bunch of dylibs will solve the real problem at hand, so this seems unlikely to be a quick bug fix in Cargo.

[zicklag]

@alexcrichton, you said in your other comment:

note that the crate type "cdylib" is most appropriate for creating a Rust DSO. If you're trying to use a dylib linking to other Rust dylibs it'll be a nonstop world of pain right now unfortunately, but cdylib should be quite smooth.

I'm sorry if this is the wrong spot to ask and I'll be happy to move this discussion somewhere else if necessary, but if you use cdylib for a crate, the only way to link to it from another Rust crate is to use extern "C" blocks and functions that must be defined manually, right? It would be a pretty big pain to have to manually bind your API to a C interface even if you were only intending on linking to it from another Rust crate.

The problem with dylibs in Rust, on the other hand, is that the Rust ABI is unstable and a dylib built with one version of Rust will be incompatible with a dylib built with another version of Rust.

What if Rust provided a way to automatically create direct "C" bindings for a Rust crate, not with the intention of having C programs link to it, but with the purpose of giving Rust a stable ABI over which to dynamically link other Rust crates.

For example, there could be:

• A new crate type that you could build crates as such as rdylib.
• A new way to import extern crates such as extern dynamic crate crate_name ( This could maybe be done away with if it could be automatically detected that the library is an rdylib )

If you build a crate as an rdylib it would be very similar to building a dylib except that it would automatically create extern "C" functions ( either literally, or just do the equivalent under the hood ) for the entire public Rust API for that crate. When including a crate using the extern dynamic crate syntax, it would essentially automatically create extern "C" blocks for the rust crate so that you could call its functions. You would probably have to build derive macros as rdylib's as well so that they could be linked into rustc while compiling the dependent crate.

As far as the developer is concerned, they don't have to manually create any bindings, and they can now safely dynamically link to other Rust libraries.

The motivation for this is the desire to be able to create a Rust plugin interface. I've attempted a tutorial which is based on using dylib's but I think that has problems in a larger context. I think I've experienced the problem where dependent crates are not built like @jsgf did. To solve that issue in the context of my idea you might be able to build all of the dependencies as rdylib's or maybe statically link them into the single generated rdylib ( if that doesn't cause issues with diamond dependencies ).

Anyway, I would appreciate your thoughts on the subject as I think that dynamic linking would be really helpful to me if there was a way to get it to work. If there is already a way to do this without changes to Cargo or Rust that I'm not aware of, that would be great. 🙂

[zicklag]

Another thought is that you might be able to implement this without modifying rustc or Cargo by creating a macro that would create the extern functions automatically by reading a library's rlib and put them in a Rust file that could be compiled to create the shared library. All of which could probably be automated by a build script.

In a similar manner, you could create a macro for importing the crate that would expand to all of the extern blocks necessary to link to it.

That would be a good way to prove out the concept.

[alexcrichton]

For designing a system of official supported dynamic libraries in Rust I'd recommend the internals/users forum rather than an old issue on the Cargo issue tracker. It's likely to definitely need rustc changes and require an RFC eventually.

[clouds56]

How should I do with cargo if I really want to build every dependencies with dylib?
I've tried to use RUSTFLAGS="-C prefer-dynamic" cargo build --release and cargo rustc --release -- -C prefer-dynamic but without luck.

[zicklag]

I don't think you can have it build every crate as a dylib. I think that will just dynamically link the standard library with -C prefer-dynamic. Also, you probably already know, but dylib is really not stable so to speak and creates libs that are specific to rustc version.

Here is some perspective that might help a little:

https://users.rust-lang.org/t/what-is-the-difference-between-dylib-and-cdylib/28847/3?u=zicklag

[clouds56]

@zicklag thanks for kindly reply.
I do know that the dylib is not compatible between rustc versions. But I still want this feature since I'm creating a game. It took long time building even if I modified a little in the engine or other subsystems. The binary is now ~200MB and would grow as more features added.
Almost no game is delivered as a single file, I could tolerated that lots of dll in the same folder, but it's hard to deliver update if the single bin file is such large.

Maybe now cargo doesn't support make every crate as dylib, could I specify RUSTFLAGS for some specific crates (of which is used in several crates that rustc complains)

[zicklag]

Ah, yes, those are pretty much the reasons that I tried to figure out how to use dylib. I create a tutorial based on my attempts to get that working and you're welcome to check it out, but I pretty much decided it probably wasn't going to work well enough. I'm not sure if it will end up helping you or not.

I know that the bevy engine had dynamic plugin loading at one point, but I don't know how it worked.

Also take a look a ABI Stable Crates which might be your best bet.

[zyansheep]

I have an idea for how dynamic linking should work but I have no idea if it is even feasible because I have no idea how dynamic linking works.
My idea is as follows: each dependency of a program is compiled separately and named with the library name, version, and sha256 hash. i.e. rand-0.8.0-5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03
When the main program is compiled, it would have in the binary a list of sha256 hashes corresponding to different dependencies.
These files would be able to be served by crates.io or some other distribution system, or downloaded with the executable. When the program runs, it looks in some predefined place for dependencies (maybe looking for paths from environment variable?) and once it finds them all, it runs.
This system would also allow for lots of flexibility when managing and selecting dependencies compared to C where your program might error if you load the wrong version.

The ultimate version of this might look like downloading a super small executable which when run looks for specific dependencies on the system and downloads them if they don't exist.

[cdecompilador]

It would be great to have at least for debug builds an option to compile all the dependencies as dynamic libs, when you have a lot of dependencies the linking time is horrible for the minimal change