WIP: rustPlatform.fetchCargoVendor: use registry's config.json for crate download urls

[TomaSajt]

Related: #377558

Draft.
Let's discuss first.

Questions:

• how would this be useful?
  • I couldn't find a public non-crates-io registry
    • the crates-io registries are
      • registry+https://github.com/rust-lang/crates.io-index
      • sparse+https://index.crates.io/
  • so this is probably mostly useful for internal registries of companies
    • but then we'd probably also need to figure out authentication
  • maybe I should host a registry myself and try it out
• should crates from different registries be placed into different directories?
  • if not, we could get duplicates, like what rustPlatform.fetchCargoVendor: allow duplicated dependencies #387337 tries to solve, but in the FOD part of the fetcher
    • what's the likelihood of this?
  • the way crates are currently stored is not the best if we wanted to have separate directories
    • we have /tarballs currently
      • this doesn't have distinction between registries
    • it would have been better to have /registry-0, /registry-1 etc
      • the index could be based on string-sorting order
    • what should we do to keep it backwards compatible?
• considering the complexity of this, should we re-add a way that uses the cargo vendor command
  • we'd need to explicitly advertise it as a fetcher to only use if you want to use non-crates.io registries
  • does cargo vendor even support other registries?
    • I haven't checked
• should we do something similar to importCargoLock instead?

  • nixpkgs/pkgs/build-support/rust/import-cargo-lock.nix

    Lines 22 to 31 in 32c543a

    	# Additional registries to pull sources from
    	# { "https://<registry index URL>" = "https://<registry download URL>"; }
    	# or if the registry is using the new sparse protocol
    	# { "sparse+https://<registry download URL>" = "https://<registry download URL>"; }
    	# where:
    	# - "index URL" is the "index" value of the configuration entry for that registry
    	# https://doc.rust-lang.org/cargo/reference/registries.html#using-an-alternate-registry
    	# - "download URL" is the "dl" value of its associated index configuration
    	# https://doc.rust-lang.org/cargo/reference/registry-index.html#index-configuration
    	extraRegistries ? { },

    • this doesn't use the config.json endpoint, it requires people to manually provide the download endpoint
    • I think this is worse than using the config.json automatically
• what can we learn from crane?
  • they already have a method to handle the config.json files

TODOs:

• override other registries in the config.toml using replace-with
  • copy how importCargoLock does it
• make separate directories in the non-FOD part
  • like with rustPlatform.fetchCargoVendor: allow duplicated dependencies #387337

---

I don't think a decision needs to be made ASAP, since this problem is pretty niche, and people can use importCargoLock anyways.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[simonzkl]

I couldn't find a public non-crates-io registry

The one I used to test an issue a while back was: https://crates.polomack.eu/ (https://github.com/Hirevo/alexandrie-index)

should crates from different registries be placed into different directories?

Crane does this for some reason, but cargo vendor does not.

considering the complexity of this, should we re-add a way that uses the cargo vendor command

I've seen a lot of pushback from nixpkgs maintainers to using upstream package managers to fetch dependencies (see yarn, pnpm, etc.). Unless there are some upstream stability guarantees for cargo vendor, I think this would not be accepted.

does cargo vendor even support other registries?

It does. I'm using it as a workaround right now (custom cargoDeps override).

should we do something similar to importCargoLock instead?

I think that's a fine approach, however for private registries we would need to be able to set custom headers to authenticate. Maybe something similar to the netrcPhase of pkgs.fetchurl where you can configure curlOpts="-H <auth-header>"? Obviously since we're not using curl this would be very different.

[TomaSajt]

does cargo vendor even support other registries?

It does. I'm using it as a workaround right now (custom cargoDeps override).

Since cargo vendor supports auth, even if not 100% guaranteed to be stable, IMO it's the best solution.
But this takes us back to the question of whether all people who need custom registries should roll their own cargo vendor-based logic, or if there should be a basic nixpkgs implementation.

As for fetchCargoVendor, I myself do not wish to implement the cargo registry authorization API logic in it.

[TomaSajt]

I couldn't find a public non-crates-io registry

The one I used to test an issue a while back was: https://crates.polomack.eu/ (https://github.com/Hirevo/alexandrie-index)

I have also since found https://github.com/rust-lang/staging.crates.io-index

Sadly, both of these seem to be git-based registry indexes.
It would be great if I could test the sparse protocol as well...

[TomaSajt]

Btw I found sparse+https://mirrors.ustc.edu.cn/crates.io-index/
just a mirror, but at least it's different...