Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Want config option for definitively specifying local crate paths #6713

Open
ijackson opened this issue Mar 3, 2019 · 5 comments

Comments

@ijackson
Copy link

commented Mar 3, 2019

Problem

It is very usual when doing development to have local unpublished dependencies. Also, when cargo is being run as part of some larger build system, the larger build system typically wants to control what dependencies are used.

cargo does not offer a good way to handle this situation. The config option paths looks like it would be good for this. However, it does not work if the crate is not in the registry (or if the metadata in the registry is not appropriate for the local version).

It is of course possible to edit Cargo.toml in each crate to specify the local pathnames of all these local crates. But this is not good:

  • It is tiresome to find and edit all the relevant Cargo.toml;
  • Some code somewhere has to read and parse the Cargo.toml's well enought to figure out where and how to add the path.
  • This approach does not work when a published crate from the registry has a local crate as a dependency (in which case the user wants to build the published crate using the local version), meaning a further workaround of adding to the set of local crates.
  • Editing Cargo.toml in every involved crate makes every involved git tree dirty - even trees that the user has no intention of modifying and just wanted a different revision of.

This last problem is quite severe. It interferes with proper use of version control. The workaround for that involves committing paths on the local computer to git. This is of course quite unclean and tends to cause these local paths to leak out of their appropriate context (which was the computer they started on).

i guess right now many people commit these local paths to git, presumably meaning to strip them out later. AIUI this has even resulted in a workaround on crates.io which according to some sources now strips path entries out of Cargo.toml during the publication process.

Proposed solution

What I think is needed is an option like paths but which is processed before the registry is consulted, before the dependency graph is processed, and so on. The information in the new config option should completely override anything in the registry or in any Cargo.tomls.

This should work even if the registry information is unparseable - that way this can be used as a workaround for corrupted information in the registry.

Ideally the new option should permit specifying the fallback behaviour on a per-crate basis (ie, what happens if for any particular purpose none of the new_path crates are suitable: either an error, or falling back to the registry).

For reference, here is the most useful stackexchange question which seems to discuss this:
https://stackoverflow.com/questions/33025887/how-to-use-a-local-unpublished-crate

Nightmarish workaround

In the meantime, I have worked around this problem with an absolutely horrific shell script.

My shell script is a wrapper for cargo. On each invocation it reads a file containing a list of crate names and paths, and edits (using sed!) all the Cargo.toml's to specify the right path. It then runs cargo. When cargo is done, it puts everything back, so your git trees are all left clean (although they were dirtied during the build).

I have attached the script, in case it is useful to anyone else. It is also here:
https://www.chiark.greenend.org.uk/ucgi/~ianmdlvl/git?p=nailing-cargo.git;a=blob;f=nailing-cargo;h=9bf28b57d21212edeefebddb52b2d637f18fe1f3;hb=HEAD

The seddery is a particularly bad feature of this script. It worked for me with the crates I cared about. Also the script is full of clone-and-hack and its fd handling is too ad-hoc. I would love to throw this thing away.

nailing-cargo.txt

@joshtriplett

This comment has been minimized.

Copy link
Member

commented Mar 4, 2019

(Note: edited this comment to expand it and make it clearer.)

Another alternative here would be the [replace] section in your crate, which you can use to replace bits of the dependency graph, and you'd only have to edit the top-level crate, not any of the dependencies. That section gets automatically get stripped out when you publish your crate to crates.io.

As another alternative, you could create your own directory registry: make a directory containing a symlink farm to the directories in /usr/share/cargo/registry, and then add symlinks in that directory to your local crates. As far as cargo is concerned, those local crates will then be in the registry.

Please let me know if one or the other of those works for you.

(The eventual thing you'll want is alternative/custom registries, which will let you have a local registry with completely different packages that gets consulted before crates.io. That isn't quite there yet, but it's close.)

@ijackson

This comment has been minimized.

Copy link
Author

commented Mar 5, 2019

@mattheww

This comment has been minimized.

Copy link

commented Mar 5, 2019

I agree with the submitter that it's worth providing a way to override Cargo.toml without editing it.

I've been using git smudge/clean filters to avoid putting local configuration in the committed Cargo.toml, and I don't think that sort of thing ought to be needed for something as common as working with a not-yet-released library.

@fanzeyi

This comment has been minimized.

Copy link

commented Aug 19, 2019

We are now facing a very similar problem as described in this issue. We need to override a local unpublished crate due to the use of different build systems.

The background is that we are building Rust with three different build systems (Cargo itself, one calls into Cargo, and the other one don't use Cargo at all -- directly uses rustc), and the one uses Cargo under the hood is re-arranging the directory structure for other purposes before invoking Cargo. This causes a problem because it would create a different directory structure comparing to the structure when we build directly with Cargo. In the Cargo.toml, we are specifying the location to that crate with relative path. Since it is an unpublished crate and specified in a form of crate = { path = "../../../path/to/crate" }, we are unable to override its path other than making the build system editing the Cargo.toml directly -- which isn't a sustainable solution and we'd love to avoid doing. (Cross-platform compatibility is also something we need so it couldn't be a shell script doing the editing :/)

I tried to use [patch] [replace], paths in .cargo/config and making a local registry as suggested in the comment above, but none of this work because it is a local non-published crate.

I want to know if the Rust/Cargo team would accept such changes to allow overriding paths to local unpublished crates. I will be in RustConf this week in Portland, and we definitely want to chat about this in person if possible.

@ijackson

This comment has been minimized.

Copy link
Author

commented Aug 20, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.