Add support for Rust / Cargo packaging #7501
Conversation
This is useful when `leaveDotGit = true` and some other derivation expects some branch name to exist. Previously, `nix-prefetch-git` always created a branch with a hard-coded name (`fetchgit`).
|
Looks cool! Going to look deeper later. Thank you! |
Also disable check phase in cargo as there are lots of failures (some probably due to trying to access the network).
|
Awesome, thanks for the hard work! There's indeed some very hackish things in there, but that's good enough for now :) We'll also need to add a paragraph in the documentation about how to package a rust package. I think we should invoke some nixpkgs guru for review, since this is very big. By the way, what do you think about dropping the rust snapshot after 1.0 is released? |
It turns out that `cargo`, with respect to registry dependencies, was ignoring the package versions locked in `Cargo.lock` because we changed the registry index URL. Therefore, every time `rustRegistry` would be updated, we'd always try to use the latest version available for every dependency and as a result the deps' SHA256 hashes would almost always have to be changed. To fix this, now we do a string substitution in `Cargo.lock` of the `crates.io` registry URL with our URL. This should be safe because our registry is just a copy of the `crates.io` registry at a certain point in time. Since now we don't always use the latest version of every dependency, the build of `cargo` actually started to fail because two of the dependencies specified in its `Cargo.lock` file have build failures. To fix the latter problem, I've added a `cargoUpdateHook` variable that gets ran both when fetching dependencies and just before building the program. The purpose of `cargoUpdateHook` is to do any ad-hoc updating of dependencies necessary to get the package to build. The use of the '--precise' flag is needed so that cargo doesn't try to fetch an even newer version whenever `rustRegistry` is updated (and therefore have to change depsSha256 as a consequence).
|
@madjar Which rust snapshot do you mean? I thought that the |
|
Note that I've just noticed that there was a bug and I've just added a fix for it. It turns out that I've also found a way to get rid of the However, the major benefit is that it would make I assume this is a worthwhile tradeoff, so if nobody objects I will start working on it tomorrow. |
|
@wizeman Oh, sorry, I meant the rustcMaster. What upstream change would be required to make it cleaner ? |
Instead, move that code into buildRustPackage. The setup hook was only doing part of the work anyway, and having it in a separate place was obscuring what was really going on.
... in a more generic way. With this commit, if you need to patch a registry package to make it work with Nix, you just need to add a script to patch-registry-deps in the same style as the `pkg-config` script.
This makes buildRustPackage portable to non-Linux platforms. Additionally, now we also save the `Cargo.lock` file into the fetch output, so that we don't have to run $cargoUpdateHook again just before building.
For some reason, `cargo` doesn't like git repositories downloaded with `fetchgit`. It will sometimes fail with the error message "Target OID for the reference doesn't exist on the repository". To workaround this, we'll just have to create a new git repo from scratch...
|
@madjar I think we shouldn't delete Regarding making this cleaner, I opened rust-lang/cargo#1334 a few months ago, but it was closed and I was told to simply rely on |
|
Ok, I've fixed a new bug that I found, where the local registry was being ignored because cargo sometimes doesn't seem to like repositories downloaded by I'm not sure why this happens sometimes, I've tried to look into it a bit but I just don't know much about git internals and I wasn't looking forward to start digging into the On the other hand, I finally got rid of the I also did a few cleanups and I added a bunch of comments explaining what is being done, especially in the Since I was on a roll, I also factored out the patching of the |
|
Very nice! Question though, did you involve any Rustaceans in your struggles with On Fri, Apr 24, 2015 at 1:22 AM Ricardo M. Correia notifications@github.com
|
I opened cargo issue 1330 and was then told to open another issue (cargo issue 1334), but in the end there didn't seem to be any interest in what I was proposing... At this point I'm not even sure what I'd like to propose anymore. Although I'm tempted to want a more traditional approach to packaging Rust libraries, such as what we do for other languages, their suggested use of The only major worry I have with the way cargo bundles Rust libraries is the way cargo / rustc wants to statically link almost all library dependencies. Specifically, I am concerned about the difficulty of ensuring we are not shipping any Rust program with a statically linked library with known security vulnerabilities. But I'm also thinking that assuming most Rust programs will depend on libraries that are in the It would be more difficult to do this for git-based dependencies, though... |
|
When are you going to merge this? |
Add support for Rust / Cargo packaging
|
Merged. |
|
Congratulations! |
As I mentioned in a comment in #4568, I continued working on Rust packaging starting from where @madjar left off.
This took me a lot longer and it was a lot harder and hackier than I thought. It's not in a completely finished and polished state, but at least it seems to work now. It allows Nix to build cargo'ed Rust programs with a minimum of fuss, including
cargoitself, and it should work in a deterministic way.Note that the
fetch-cargo-depsscript is probably not very robust yet with respect to git checkouts.Specifically, I have not tested building a Rust program that has a git dependency whose repository has submodules (I would be surprised if this worked without some additional fixes). If anyone has an example of such a program, please let me know and I will fix the script.
As @madjar had mentioned, it would be nice to add a
nix-prefetch-cargoscript and documentation, but even in this state I think it would already be useful to anyone who wants to build Rust programs.A big thanks to @madjar for his initial work!