Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Git sources of child crates are not properly used #33

Closed
andir opened this issue Oct 9, 2019 · 3 comments
Closed

Git sources of child crates are not properly used #33

andir opened this issue Oct 9, 2019 · 3 comments
Labels
before-1.0 This needs to be worked on/fixed before a 1.0. bug Something isn't working
Milestone

Comments

@andir
Copy link
Collaborator

andir commented Oct 9, 2019

When building a package from a git source where the git repository contains multiple crates the nix expression (sometimes?) doesn't know about the child directory for dependency in relative child directories.

Example:

Imagine the following section in your Cargo.toml:

[dependencies.rocksdb]
git = "https://github.com/rust-rocksdb/rust-rocksdb"
rev = "bc8520a86e96c38f79caac0d09349a01cbd113e9"
branch = "0.12.2"
default-features = false
features = ["lz4"]

Then looking further into the Cargo.toml of the rocksdb crate:

[dependencies]
libc = "0.2"
librocksdb-sys = { path = "librocksdb-sys", version = "5.18.3" }

(https://github.com/rust-rocksdb/rust-rocksdb/blob/bc8520a86e96c38f79caac0d09349a01cbd113e9/Cargo.toml#L26-L28)

We see that the librocksdb-sys crate is supposed to be at the relative path librocksdb-sys within the rocksdb crate source. When we now look at the definition of librocksdb-sys in the generated Cargo.nix we see the following:

      "librocksdb-sys …"
        = rec {
          crateName = "librocksdb-sys";
          version = "5.18.3";                                                                                                                                                                                                                  
          edition = "2015";                                                                                                                                                                                                                    
          src = builtins.fetchgit {                                                                                                                                                                                                                    
            url = "https://github.com/rust-rocksdb/rust-rocksdb";                                                                                                                                                                        
            rev = "bc8520a86e96c38f79caac0d09349a01cbd113e9";                                                                                                                                                                                  
            sha256 = "…";                                                                                                                                                                   
          };                                                                                                                                                                                                              
          authors = [];                                                                                                                                                                                                                                   
          dependencies = {};                                                                                                                                                                                                                                   
          buildDependencies = {};                                                                                                                                                                                                                                   
          features = {                                                                                                                                                                                                                         
            "default" = [ "static" ];                                                                                                                                                                                                          
          };                                                                                                                                                                                                                                   
          resolvedDefaultFeatures = [ "default" "lz4" "static" ];                                                                                                                                                                              
        };                   

What is missing here is the child directory that actually hosts the crate librocksdb-sys. By changing the src attribute to

src = (builtins.fetchgit {                                                                                                                                                                                                                    
            url = "https://github.com/rust-rocksdb/rust-rocksdb";                                                                                                                                                                        
            rev = "bc8520a86e96c38f79caac0d09349a01cbd113e9";                                                                                                                                                                                  
            sha256 = "…";                                                                                                                                                                   
          }) + "/librocksdb-sys";

we can workaround the issue.

I spent some time looking at how cargo handles those internally. It seems like each directory is cloned into your local "db" (in cargo speech?) at ~/.cargo/git/checkouts/<repo>-<somehash>/<git short ref>/. In the above example the package_path of our package then is ~/.cargo/git/checkouts/<repo>-<somehash>/<git short ref>/librocksdb-sys/.

The choices we have here seem to be limited. The trivial approach to fixing this would be to collect path elements from the package_path until the short id has been found. The collected paths can then be used as a suffix of the git cloned sources.

@andir andir changed the title git sources of crate Git sources of child crates are not properly used Oct 9, 2019
@kolloch kolloch added bug Something isn't working before-1.0 This needs to be worked on/fixed before a 1.0. labels Oct 11, 2019
@kolloch
Copy link
Collaborator

kolloch commented Oct 11, 2019

Thank you for the excellent description of the problem and the analysis! I will look if I have enough info in the cargo metadata output to fix this easily.

@kolloch
Copy link
Collaborator

kolloch commented Nov 27, 2019

Solving #53 might also solve this.

@kolloch kolloch closed this as completed Nov 27, 2019
@kolloch kolloch reopened this Nov 27, 2019
@kolloch kolloch added this to the 0.8 milestone Feb 12, 2020
kolloch added a commit to kolloch/nixpkgs that referenced this issue Mar 9, 2020
…sub directories

This is what cargo does for git repositories.

See related issues:

* nix-community/crate2nix#53
* nix-community/crate2nix#33
kolloch added a commit that referenced this issue Mar 11, 2020
Cargo does this for git repositories as well:

rust-lang/cargo#1462 (comment)

This will fix #33, #53 -- integration tests pending.

It depends on a nixpkgs buildRustCrate feature PR:

NixOS/nixpkgs#82155
kolloch added a commit that referenced this issue Mar 11, 2020
Added tests for real crates depending on workspace_member = null.
@kolloch
Copy link
Collaborator

kolloch commented Mar 11, 2020

I verified my fix against this with an integration tests. I wait until the relevant PR (See #53) hits nixpkgs-unstable. I close this and track this further in #53.

@kolloch kolloch closed this as completed Mar 11, 2020
kolloch added a commit that referenced this issue Mar 11, 2020
Added tests for real crates depending on workspace_member = null.
kolloch added a commit that referenced this issue Mar 12, 2020
Cargo does this for git repositories as well:

rust-lang/cargo#1462 (comment)

This will fix #33, #53 -- integration tests pending.

It depends on a nixpkgs buildRustCrate feature PR:

NixOS/nixpkgs#82155
kolloch added a commit that referenced this issue Mar 12, 2020
Added tests for real crates depending on workspace_member = null.
kolloch added a commit that referenced this issue Mar 12, 2020
Cargo does this for git repositories as well:

rust-lang/cargo#1462 (comment)

This will fix #33, #53 -- integration tests pending.

It depends on a nixpkgs buildRustCrate feature PR:

NixOS/nixpkgs#82155
kolloch added a commit that referenced this issue Mar 12, 2020
Added tests for real crates depending on workspace_member = null.
andir pushed a commit to andir/nixpkgs that referenced this issue Mar 28, 2020
…sub directories

This is what cargo does for git repositories.

See related issues:

* nix-community/crate2nix#53
* nix-community/crate2nix#33

(cherry picked from commit 8a6638d)
stigok pushed a commit to stigok/nixpkgs that referenced this issue Jun 12, 2020
…sub directories

This is what cargo does for git repositories.

See related issues:

* nix-community/crate2nix#53
* nix-community/crate2nix#33

(cherry picked from commit 8a6638d)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
before-1.0 This needs to be worked on/fixed before a 1.0. bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants