Intelligent handling of native library dependencies

[djrenren]

The Problem

Many crates involve a C library somewhere in their dependency chain.

Solutions

1. Rely on each sys package to handle native dependencies

Pros

• "Simplest" solution. Keeps cargo simple.

Cons

• Inconsistent handling of library dependencies makes users sad 😢
• Lots of duplicated logic.

2. Create a utility library to handle native dependencies in build.rs

Pros

• Keeps cargo simple
• Reasonably consistent dependency handling. (Assuming general adoption)
• Capabilities can evolve more rapidly to community needs

Cons

• Including prebuilts in the package for every platform is obviously going to be too large.
• Fetching libraries is really difficult:
  • No http or ftp implementation provided by default
  • Https fetching would rely on a native package in the first place
    Would duplicate the responsibility of fetching dependencies between cargo and the packages.

3. Let Cargo handle C library dependencies

Pros

• Using platform-specific dependencies, packages would only pull down prebuilt binaries for the specific platform
• Less work for sys package makes
• Standard sys package behavior
• Can fetch dependencies well for cross-compilation

Cons

• Detecting library presence is difficult
• Complicates Cargo
• Might require http/https/ftp (or even zip/tgz) implementation in cargo for fetching prebuilts
• Dynamic linking gets messy

As is probably obvious, I'm a proponent of the 3rd option. I think it would be the best for the rust community, especially those on Windows. Windows will almost always be lacking these libraries and installing them into rust's msys install is always going to be painful so having a standard procedure would be a huge step toward alleviating this.

[alexcrichton]

One of the major reasons for creating build scripts was to share dependencies among build scripts, so I think that option 1 is basically out of the picture in terms of "where we want to be". I do understand, however, that it takes time to create high-quality crates for Option 2 both with respect of ease of use and flexibility. There doesn't necessarily have to be "one true" library as well though! I for example have separate repositories for compiling custom C code and running pkg-config.

In my mind the compilation of Cargo in option 3 is a very large drawback. Any functionality we would provide in Cargo would essentially be a duplicate of what could already be provided in a build-dependency of a build script today.

All in all, I do agree that writing build scripts today is still a little painful due to lack of build dependencies, and this could just mean that we need to foster discovery of existing build dependencies. In general though I've been quite happy moving as much build logic out of Cargo and into crates themselves and in general it seems to be working out quite well.

Do you think that option 3 enables some use cases that are not enabled by option 2?

[djrenren]

First off, thanks for the incredibly thoughtful reply.

In terms of use cases, I think you're right that there wouldn't be much to distinguish 2 and 3.

The ability to pull dependencies from non-git sources would be nice especially in environments where internal projects may be required but not hosted on git. Though the complexity involved is probably not worth it at this point.

Ultimately with proper tooling and community support, build dependencies could solve the problem. I'm just worried about getting community support behind the idea of crates ensuring the availability of their libraries. At this point most do not and include a readme entry about installing manually. This may be fixed with proper tooling, but I figured inclusion of tools in cargo might make a statement of support for this behavior.

I see your reasoning and thinking about it down the line, I agree it's best to solve this with build dependencies.

[alexcrichton]

The ability to pull dependencies from non-git sources would be nice especially in environments where internal projects may be required but not hosted on git.

I will say that it's was somewhat always envisioned that Cargo will one day grow support for other VCS systems (e.g. mercurial or SVN). Right now we just happen to only have had time to write bindings for git :)

This may be fixed with proper tooling, but I figured inclusion of tools in cargo might make a statement of support for this behavior.

I definitely agree that "official solutions" are excellent catalysts for disseminating information and promoting good practices. I think the problem is that right now we ourselves do not have a great solution! I think that we may want to more heavily promote build dependencies and make them more easily discoverable.

You've hit the nail on the head about how build scripts are tough to write in robust fashions and it's easier right now to just largely punt on them entirely.

For now though I don't think that we want to start including much support in Cargo itself, so I'm going to close this. I've opened a separate issue, about promoting build dependencies: #1251.

[alexcrichton]

Also, I should also say thank you for taking time out to write such a detailed issue, I know I certainly always appreciate it!