rustPlatform.importCargoLock does not support some 3rd party registries

[djacu]

The Cargo Book specifies what the crate download path should look like, however, not all registries follow this specification. Assumptions in importCargoLock include:

• The registry endpoint will have .tar.gz files available. Some only supply .crate files.
• The path to the crate file is <dl endpoint>/<name>/<version>/crate-<name>-<version>.<ext>. This is not generally true.

The patch below demonstrates the changes I have made to a vendored version of import-cargo-lock.nix in order to support 3rd party registries that do not follow the same pattern as crates.io. This moves part of the logic from fetchCrate into the registries attribute. Each entry in registries must now provide a fetch function that must take a pkg and checksum argument and return fetched crate derivation. This allows end users to add their own registries via extraRegistries with a fetching function that works for whatever registry they use.

---
 overlays/rust-platform/import-cargo-lock.nix | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)

diff --git a/overlays/rust-platform/import-cargo-lock.nix b/overlays/rust-platform/import-cargo-lock.nix
index 8d3391e5d1bc..7346bec9893c 100644
--- a/overlays/rust-platform/import-cargo-lock.nix
+++ b/overlays/rust-platform/import-cargo-lock.nix
@@ -120,7 +120,7 @@ let
   # We can't use the existing fetchCrate function, since it uses a
   # recursive hash of the unpacked crate.
   fetchCrate =
-    pkg: downloadUrl:
+    pkg: registry:
     let
       checksum =
         pkg.checksum or parsedLockFile.metadata."checksum ${pkg.name} ${pkg.version} (${pkg.source})";
@@ -128,14 +128,20 @@ let
     assert lib.assertMsg (checksum != null) ''
       Package ${pkg.name} does not have a checksum.
     '';
-    fetchurl {
-      name = "crate-${pkg.name}-${pkg.version}.tar.gz";
-      url = "${downloadUrl}/${pkg.name}/${pkg.version}/download";
-      sha256 = checksum;
-    };
+    registry.fetch pkg checksum;
 
   registries = {
-    "https://github.com/rust-lang/crates.io-index" = "https://crates.io/api/v1/crates";
+    "https://github.com/rust-lang/crates.io-index" = rec {
+      downloadUrl = "https://crates.io/api/v1/crates";
+      fetch =
+        pkg: checksum:
+        fetchurl {
+          name = "crate-${pkg.name}-${pkg.version}.tar.gz";
+          url = mkPkgUrl pkg;
+          sha256 = checksum;
+        };
+      mkPkgUrl = pkg: "${downloadUrl}/${pkg.name}/${pkg.version}/download";
+    };
   } // extraRegistries;
 
   # Replaces values inherited by workspace members.
-- 
2.47.1

The following is a separate issue but related by regarding crates that have dependencies in 3rd party registries. A build failure occurs in a specific situation where you are trying to build a package that has a direct or transitive git dependency that has a dependency that is in a 3rd party registry. I.e.

graph LR;
    A[my package];
    B[git dependency];
    C[3rd party registry];
    A-->B;
    B-->C;

Loading

The issue comes from the mkCrate command in import-cargo-lock.nix. There are two times cargo metadata is called from the sandbox but targets a manifest file somewhere in the nix store, $tree/Cargo.toml. This path is a dependency of the derivation being built. It does have a .cargo/config.toml file with the 3rd party registry information, however, cargo will not discover this file. This is because cargo only searches for the config.toml file in the current directory up to the root filesystem and in $CARGO_HOME. Below is a solution to the problem which runs cd $tree in the subshell before running cargo metadata. It might not be the most elegant or correct but simply illustrates how the problem could be solved.

As a side note, this issue is not present for dependencies from a 3rd party registry that have dependencies that also using the registry. This is because when crates are published, cargo will replace the Cargo.toml file with an auto-generated one that inlines registry index information from the config.toml file.

---
 overlays/rust-platform/import-cargo-lock.nix | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/overlays/rust-platform/import-cargo-lock.nix b/overlays/rust-platform/import-cargo-lock.nix
index 7346bec9893c..c38f07f3cc14 100644
--- a/overlays/rust-platform/import-cargo-lock.nix
+++ b/overlays/rust-platform/import-cargo-lock.nix
@@ -216,7 +216,7 @@ let
         # but only in nested directories.
         # Only check the top-level Cargo.toml, if it actually exists
         if [[ -f $tree/Cargo.toml ]]; then
-          crateCargoTOML=$(${cargo}/bin/cargo metadata --format-version 1 --no-deps --manifest-path $tree/Cargo.toml | \
+          crateCargoTOML=$(cd $tree && ${cargo}/bin/cargo metadata --format-version 1 --no-deps --manifest-path $tree/Cargo.toml | \
           ${jq}/bin/jq -r '.packages[] | select(.name == "${pkg.name}") | .manifest_path')
         fi
 
@@ -224,7 +224,7 @@ let
         if [[ -z $crateCargoTOML ]]; then
           for manifest in $(find $tree -name "Cargo.toml"); do
             echo Looking at $manifest
-            crateCargoTOML=$(${cargo}/bin/cargo metadata --format-version 1 --no-deps --manifest-path "$manifest" | ${jq}/bin/jq -r '.packages[] | select(.name == "${pkg.name}") | .manifest_path' || :)
+            crateCargoTOML=$(cd $tree && ${cargo}/bin/cargo metadata --format-version 1 --no-deps --manifest-path "$manifest" | ${jq}/bin/jq -r '.packages[] | select(.name == "${pkg.name}") | .manifest_path' || :)
             if [[ ! -z $crateCargoTOML ]]; then
               break
             fi
-- 
2.47.1