Can serde support crate-type dylib?

[rksm]

This might be a bit of a long shot, but I thought I would ask :)

Dynamically linking crates can improve the incremental compilation time of projects substantially. serde, as a dependency that can increase compilation times quite a bit, would be a good target to make dynamic.

Since serde is usually used in combination with crates that implement specific serialization formats such as serde_json it'll normally be added as a dependency by both the final user and the crate implementing the serialization format like serde_json.

When wrapping both serde and serde_json as dylibs to improve compilation speed, we end up with a situation like

  graph TD;
      bin["binary"]-.->serde_dylib["serde (dylib)"];
      bin-.->serde_json_dylib["serde_json (dylib)"];
      serde_json_dylib-->serde_json;
      serde_json-->serde1["serde"];
      serde_dylib-->serde2["serde"];

Loading

Since serde does not specify crate-type dylib it'll need to be linked statically by both serde_json and the dynamic serde wrapper. This is something that is not supported by Rust: rust-lang/rust#34909 (comment). We basically run into the diamond dependency problem, where two static copies of serde end up in the dependency graph and Rust errors with error: cannot satisfy dependencies so serde only shows up once as it cannot decide which copy to use. This is also explained in the module comment of rust/compiler/rustc_metadata/src/dependency_format.rs.

A solution to the problem would be to allow dynamic linking against serde by adding

[lib]
crate-type = ["rlib", "dylib"]

to serde/Cargo.toml. This would let consumers of the crate allow to opt-in to dynamic linking, basically allowing for a dependency graph like

  graph TD;
      bin["binary"]-.->serde["serde (dylib)"];
      bin-.->serde_json_dylib["serde_json (dylib)"];
      serde_json_dylib-->serde_json;
      serde_json-.->serde;

Loading

This would not change how serde is linked if crates do not require dynamic linking as static linking is the default (see dependency_format.rs). So this should be a backwards compatible change.
Only if serde is used like in the initial example, this would allow it to be used as a dylib. RUSTFLAGS="-C prefer-dynamic" would have the same effect.

This issue has been raised here originally.

I'm aware of the importance of serde and know that such a change cannot be done lightly. So if this is asking too much, feel free to close. On the other hand, I think it would have quite a positive impact on incremental compilation time and is worth bringing up.

[dtolnay]

Cargo's support for dylib is not suitable for this, regardless of any code change in serde. Nothing in what you've described is specific to serde or serde_json — you're just describing arbitrary crate A which depends on arbitrary crate B and cargo's dylib is not designed to handle it.

I would recommend looking into a different build system that is better suited to building dependency trees using dylibs (Bazel comes to mind).

[rksm]

Thank you for your response. I'm aware that dyilb support in Cargo has not been a focus for the teams and that there are likely a number of issues lurking. Just that the above seems to work fine – allowing serde to be build as dylib and requiring it through another dylib will demonstrably do the right thing, at least in my test setup.

If I understand correctly, you are saying that there is no guarantee that cargo will structure the dependency tree in a way that shared libraries will get used in all cases correctly? I went through the code computing the dep sets and there is nothing that would obviously not work. Do you have some more pointers at where this approach might fail?

[crazyboycjr]

@dtolnay I'm experiencing a similar diamond dependency issue which I described here. Do you think it would make sense to you if I make all transitive dependencies dylib and link them together in my project (by customizing cargo a little bit)? I'm not sure whether making everything dylib is the right thing to do and would like to hear your suggestions. Thanks so much!