Use crate2nix generate for third-party projects

[flokli]

I ran into another FOD-related issue when packaging some rust project, and would really like to be able to use crate2nix to package "external" projects.

While it's possible to invoke crate2nix generate with a -f pointing to an external path, the underlying attrset inside internal has a src pointing to some weird src (in my case, a relate path up to my /, and then descending down into /nix/store, which will make the Cargo.nix file not portable.
This is only the case for the crate/workspace that the Cargo.nix is created for, not dependencies.

While this approach works fine for monorepos, when crate2nix generate is executed alongside the sources, and the file is staying there, it doesn't work good for "external projects".

It'd be nice if crate2nix generate had some "external project mode", in which it does not write src for the root crate/workspace, but asks for a mandatory toplevel src argument.

[kolloch]

https://github.com/nix-community/crate2nix/blob/master/out-of-tree-sources.md Maybe this helps?

[flokli]

I don't think so. The crate2nix source add invocations just imperatively assemble a list of dependencies, and the section further down describes the crate2nix generate process I tried to use.

This issue is explicitly about packaging something for which I don't have the checkout locally, but pull sources via Nix (via fetchFromGitHub for example).

A more graspable and similar example (even though that's not the goal here) would be "have a mode of operation to use crate2nix to generate a Cargo.nix from a third-party repo" - assuming we could commit Cargo.nix into nixpkgs, and use it with a nix-provided src.

[kolloch]

Well, the title is: "Handling out-of-tree sources"

Even though it is a while ago that I wrote this, the intention was exactly supporting "out of tree" sources as you describe.

The examples add binaries which sources are fetches from cratesIo. But it can also fetch from github as mentioned.

❯ mkdir thirdpart-binary-workspace
❯ cd thirdpart-binary-workspace 
❯ nix run github:nix-community/crate2nix -- source add git https://github.com/ClementTsang/bottom --rev 712a0036811d1ec1f676b2ba76fff4bf5cc5fa97
Prefetching https://github.com/ClementTsang/bottom#712a0036811d1ec1f676b2ba76fff4bf5cc5fa97: done.
Added new source: https://github.com/ClementTsang/bottom#712a0036811d1ec1f676b2ba76fff4bf5cc5fa97 via git: 1p12lay9haj4210am2f6d9mj0f7srmg7sm8pljns92a7xnrsp7zn
❯ ls -al   
total 12
drwxrwxr-x  2 peter peter 4096 Okt 31 18:01 .
drwxrwxr-x 17 peter peter 4096 Okt 31 17:59 ..
-rw-r--r--  1 peter peter  248 Okt 31 18:01 crate2nix.json
❯ cat crate2nix.json 
{
  "sources": {
    "bottom": {
      "type": "Git",
      "url": "https://github.com/ClementTsang/bottom",
      "rev": "712a0036811d1ec1f676b2ba76fff4bf5cc5fa97",
      "sha256": "1p12lay9haj4210am2f6d9mj0f7srmg7sm8pljns92a7xnrsp7zn"
    }
  }
}%                                                                                                                                                                                     
❯ nix run github:nix-community/crate2nix -- generate                                                                                            
Fetching sources.
Generated ./crate2nix-sources.nix successfully.
Fetching sources via ./crate2nix-sources.nix fetchedSources: done.
Generated ./Cargo.nix successfully.

❯ ls
Cargo.nix  crate2nix.json  crate2nix-sources  crate2nix-sources.nix

❯ cd crate2nix-sources 

❯ ls -al
total 940
dr-xr-xr-x    2 root root     4096 Jan  1  1970 .
drwxrwxr-t 1565 root nixbld 954368 Okt 31 18:02 ..
lrwxrwxrwx    2 root root       58 Jan  1  1970 bottom -> /nix/store/9fvkl41sjkmc42l7nkr7hzpjljd8aapn-bottom-712a003
❯ cd ..
❯ ls -al
total 252
drwxrwxr-x  2 peter peter   4096 Okt 31 18:02 .
drwxrwxr-x 17 peter peter   4096 Okt 31 17:59 ..
-rw-r--r--  1 peter peter 237461 Okt 31 18:02 Cargo.nix
-rw-r--r--  1 peter peter    248 Okt 31 18:01 crate2nix.json
lrwxrwxrwx  1 peter peter     61 Okt 31 18:02 crate2nix-sources -> /nix/store/s4bx9vqlkvng94fd5i702b777nbk8pwf-crate2nix-sources
-rw-r--r--  1 peter peter   3879 Okt 31 18:02 crate2nix-sources.nix

You can .gitignore that symlink, it is only needed during generation of Cargo.nix and automatically recreated.

❯ rm crate2nix-sources
❯ ls -al
total 248
drwxrwxr-x  2 peter peter   4096 Okt 31 18:03 .
drwxrwxr-x 17 peter peter   4096 Okt 31 17:59 ..
-rw-r--r--  1 peter peter 237461 Okt 31 18:02 Cargo.nix
-rw-r--r--  1 peter peter    248 Okt 31 18:01 crate2nix.json
-rw-r--r--  1 peter peter   3879 Okt 31 18:02 crate2nix-sources.nix
❯ nix build -f Cargo.nix workspaceMembers.bottom 
[1/213/214 built, 3 copied (0.0/0.0 MiB), 0.0 MiB DL] building rust_bottom-0.9.6 (buildPhase): Running rustc --crate-name bottom src/lib.rs --out-dir target/lib -L dependency=target/derror: interrupted by the user

[flokli]

I'm not sure what workflow I had in mind here, but that process indeed works. Sorry for the noise, and thanks :-)

[kolloch]

I was also confused by the workflow when looking at it again after years 😊

a nicer workflow could be possible with IFD or dynamic derivations…

[flokli]

IFD makes it harder to debug the emitted intermediate nix code, and dynamic derivations are still experimental (and I'm not convinced I like them ;-) ). Anyways :-D

[flokli]

I ran into this again today! Opened #316 with a reproducer, using the workflow described here.