Can't run cargo metadata --offline with crates.io replaced with a git registry

[xcthulhu]

Problem

In an attempt to resolve nix-community/crate2nix#148, I am trying to generate cargo metadata jsons using cargo metadata --offline. I need to run this using nix derivations, however, which run in sandbox environments without internet access. As a consequence I have not been able to run cargo vendor or cargo local-registry.

Instead, I have tried using a local mirror of rust-lang/crates.io-index, and specified in .cargo/config.toml to replace crates.io with that mirror. However, while crates.io supports checksums, my local mirror does not.

Steps

1. Locally mirror https://github.com/rust-lang/crates.io-index:

(mkdir /tmp/local-crates.io-index; cd /tmp/local-crates.io-index; git init --bare)
git clone https://github.com/rust-lang/crates.io-index
cd crates.io-index/
git remote add local-mirror file:///tmp/local-crates.io-index
git push local-mirror master

2. Create a project with a dependency

cargo init test-project
cd test-project

cat > Cargo.toml <<<EOF
[package]
name = "test-project"
edition = "2018"
version = "0.0.1"

[dependencies]
rayon = "1.4.0"
EOF

cargo generate-lockfile

3. Configure cargo to replace crates.io with the local mirror

mkdir .cargo

cat >.cargo/config.toml <<EOF
[source.crates-io]
replace-with = "nix-crates-registry"

[source.nix-crates-registry]
git = "file:///tmp/local-crates.io-index"
EOF

4. Run cargo metadata --offline:

cargo metadata --offline
error: failed to get `rayon` as a dependency of package `test-project v0.0.1 (/tmp/test-project)`

Caused by:
  failed to load source for dependency `rayon`

Caused by:
  Unable to update registry `https://github.com/rust-lang/crates.io-index`

Caused by:
  cannot replace `crates-io` with `nix-crates-registry`, the source `crates-io` supports checksums, but `nix-crates-registry` does not

a lock file compatible with `crates-io` cannot be generated in this situation

Notes

Output of cargo version: cargo 1.45.1 (f242df6ed 2020-07-22)

[ehuss]

I think there might be some confusion, a git source shouldn't point to an index. A git source should point to a repository that contains Cargo packages (that is, directories that contain Cargo.toml files with their sources). If you want to point to an alternate index, I think you can specify registry = "file:///tmp/local-crates.io-index". I think there are some performance problems with that (it runs very slowly), and it doesn't work "offline" (it doesn't know that a file URL doesn't require a network), but it would still require downloading all the dependencies, so I don't think it is possible to do that completely offline.

I'm a bit unclear why you can access the network to clone the index, but then can't access it to do other operations? If you can access the network beforehand, I would suggest running cargo fetch to download all the dependencies, and then you can run in --offline (no need to replace sources).

[balsoft]

it would still require downloading all the dependencies, so I don't think it is possible to do that completely offline.

Why would it need to download all the dependencies to display the metadata? Isn't a registry enough?

I'm a bit unclear why you can access the network to clone the index, but then can't access it to do other operations?

That's because the index is a single location, the sha256 of which can be easily pinned. That can't be said about output of arbitrary cargo operations with the network involved -- this would mean that every time I change dependencies of my project, I would have to change some checksum in Nix sources, which is what the issue above is trying to avoid.

[ehuss]

Isn't a registry enough?

It is not. There is data in Cargo.toml that is not sufficiently tracked in the index.

[balsoft]

Can you explain what that data is or at least point me at somewhere I can look for it?

[ehuss]

There is a lot of information about a package displayed in the metadata output (documented in https://doc.rust-lang.org/cargo/commands/cargo-metadata.html) which is not in the index (documented in https://doc.rust-lang.org/cargo/reference/registries.html#index-format). There are some other fields, like proc_macro which are also not in the index, links which is not available for old packages, and I want to say there are others, but I cannot remember.

[balsoft]

Ok, thanks!

[balsoft]

I guess the question then boils down to whether adding an "offline" mode (one where only information from the index is displayed) to cargo-metadata would be acceptable. Or, I guess, we could fetch all of the sources on Nix side, place them somewhere Cargo can find them and then run cargo metadata.

[balsoft]

I have ended up going with the second solution

Or, I guess, we could fetch all of the sources on Nix side, place them somewhere Cargo can find them and then run cargo metadata.

https://github.com/kolloch/crate2nix/pull/151/files#diff-4f1d720cc1418b9c4d0ccec33fbe7dc2R34

[xcthulhu]

It looks like @balsoft's solution is the way to go here. Closing.