Add support local mirrors of registries, take 2 #2857

URL parsing now returns a `CargoResult` and also change a few `unwrap()` calls to returning a `CargoResult` when decoding various bits and pieces of information.

Should help easily mapping packages from one source to another

This commit implements a scheme for .cargo/config files where sources can be redirected to other sources. The purpose of this will be to override crates.io for a few use cases: * Replace it with a mirror site that is sync'd to crates.io * Replace it with a "directory source" or some other local source This major feature of this redirection, however, is that none of it is encoded into the lock file. If one source is redirected to another then it is assumed that packages from both are exactly the same (e.g. `foo v0.0.1` is the same in both location). The lock file simply encodes the canonical soure (e.g. crates.io) rather than the replacement source. In the end this means that Cargo.lock files can be generated from any replacement source and shipped to other locations without the lockfile oscillating about where packages came from. Eventually this support will be extended to `Cargo.toml` itself (which will be encoded into the lock file), but that support is not implemented today. The syntax for what was implemented today looks like: # .cargo/config [source.my-awesome-registry] registry = 'https://example.com/path/to/index' [source.crates-io] replace-with = 'my-awesome-registry' Each source will have a canonical name and will be configured with the various keys underneath it (today just 'registry' and 'directory' will be accepted). The global `crates-io` source represents crates from the standard registry, and this can be replaced with other mirror sources. All tests have been modified to use this new infrastructure instead of the old `registry.index` configuration. This configuration is now also deprecated and will emit an unconditional warning about how it will no longer be used in the future. Finally, all subcommands now use this "source map" except for `cargo publish`, which will always publish to the default registry (in this case crates.io).

This commit changes how lock files are encoded by checksums for each package in the lockfile to the `[metadata]` section. The previous commit implemented the ability to redirect sources, but the core assumption there was that a package coming from two different locations was always the same. An inevitable case, however, is that a source gets corrupted or, worse, ships a modified version of a crate to introduce instability between two "mirrors". The purpose of adding checksums will be to resolve this discrepancy. Each crate coming from crates.io will now record its sha256 checksum in the lock file. When a lock file already exists, the new checksum for a crate will be checked against it, and if they differ compilation will be aborted. Currently only registry crates will have sha256 checksums listed, all other sources do not have checksums at this time. The astute may notice that if the lock file format is changing, then a lock file generated by a newer Cargo might be mangled by an older Cargo. In anticipation of this, however, all Cargo versions published support a `[metadata]` section of the lock file which is transparently carried forward if encountered. This means that older Cargos compiling with a newer lock file will not verify checksums in the lock file, but they will carry forward the checksum information and prevent it from being removed. There are, however, a few situations where problems may still arise: 1. If an older Cargo takes a newer lockfile (with checksums) and updates it with a modified `Cargo.toml` (e.g. a package was added, removed, or updated), then the `[metadata]` section will not be updated appropriately. This modification would require a newer Cargo to come in and update the checksums for such a modification. 2. Today Cargo can only calculate checksums for registry sources, but we may eventually want to support other sources like git (or just straight-up path sources). If future Cargo implements support for this sort of checksum, then it's the same problem as above where older Cargos will not know how to keep the checksum in sync

Add an abstraction over which the index can be updated and downloads can be made. This is currently implemented for "remote" registries (e.g. crates.io), but soon there will be one for "local" registries as well.

This flavor of registry is intended to behave very similarly to the standard remote registry, except everything is contained locally on the filesystem instead. There are a few components to this new flavor of registry: 1. The registry itself is rooted at a particular directory, owning all structure beneath it. 2. There is an `index` folder with the same structure as the crates.io index describing the local registry (e.g. contents, versions, checksums, etc). 3. Inside the root will also be a list of `.crate` files which correspond to those described in the index. All crates must be of the form `name-version.crate` and be the same `.crate` files from crates.io itself. This support can currently be used via the previous implementation of source overrides with the new type: ```toml [source.crates-io] replace-with = 'my-awesome-registry' [source.my-awesome-registry] local-registry = 'path/to/registry' ``` I will soon follow up with a tool which can be used to manage these local registries externally.

This flavor of source is intended to behave like a local registry except that its contents are unpacked rather than zipped up in `.crate` form. Like with local registries the only way to use this currently is via the `.cargo/config`-based source replacement currently, and primarily only to replace crates.io or other registries at the moment. A directory source is simply a directory which has many `.crate` files unpacked inside of it. The directory is not recursively traversed for changes, but rather it is just required that all elements in the directory are themselves directories of packages. This format is more suitable for checking into source trees, and it still provides guarantees around preventing modification of the original source from the upstream copy. Each directory in the directory source is required to have a `.cargo-checksum.json` file indicating the checksum it *would* have had if the crate had come from the original source as well as all of the sha256 checksums of all the files in the repo. It is intended that directory sources are assembled from a separately shipped subcommand (e.g. `cargo vendor` or `cargo local-registry`), so these checksum files don't have to be managed manually. Modification of a directory source is not the intended purpose, and if a modification is detected then the user is nudged towards solutions like `[replace]` which are intended for overriding other sources and processing local modifications.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support local mirrors of registries, take 2 #2857

Add support local mirrors of registries, take 2 #2857

Commits on Aug 1, 2016