Import the cargo-vendor subcommand into Cargo

[alexcrichton]

This commit imports the external alexcrichton/cargo-vendor
repository into Cargo itself. This means it will no longer be
necessary to install the cargo-vendor subcommand in order to vendor
dependencies. Additionally it'll always support the latest feature set
of Cargo as it'll be built into Cargo!

All tests were imported as part of this commit, but not all features
were imported. Some flags have been left out that were added later in
the lifetime of cargo vendor which seem like they're more questionable
to stabilize. I'm hoping that they can have separate PRs adding their
implementation here, and we can make a decision of their stabilization
at a later date.

The current man page for cargo vendor -h will look like:

cargo-vendor
Vendor all dependencies for a project locally

USAGE:
cargo vendor [OPTIONS] [--] [path]

OPTIONS:
-q, --quiet                    No output printed to stdout
    --manifest-path <PATH>     Path to Cargo.toml
    --no-delete                Don't delete older crates in the vendor directory
-s, --sync <TOML>...           Additional `Cargo.toml` to sync and vendor
    --respect-source-config    Respect `[source]` config in `.cargo/config`
-v, --verbose                  Use verbose output (-vv very verbose/build.rs output)
    --color <WHEN>             Coloring: auto, always, never
    --frozen                   Require Cargo.lock and cache are up to date
    --locked                   Require Cargo.lock is up to date
-Z <FLAG>...                   Unstable (nightly-only) flags to Cargo, see 'cargo -Z help' for details
-h, --help                     Prints help information

ARGS:
<path>    Where to vendor crates (`vendor` by default)

This cargo subcommand will vendor all crates.io and git dependencies for a
project into the specified directory at `<path>`. After this command completes
the vendor directory specified by `<path>` will contain all remote sources from
dependencies specified. Additionally manifest beyond the default one can be
specified with the `-s` option.

The `cargo vendor` command will also print out the configuration necessary
to use the vendored sources, which when needed is then encoded into
`.cargo/config`.

Since this change is not importing 100% of the functionality of the
existing cargo vendor this change does run a risk of being a breaking
change for any folks using such functionality. Executing cargo vendor
will favor the built-in command rather than an external subcommand,
causing unimplemented features to become errors about flag usage.

[rust-highfive]

r? @nrc

(rust_highfive has picked a reviewer for you, use r? to override)

[alexcrichton]

Note that I suspect we can't land this necessarily as-is, but I wanted to get the conversation started!

@ehuss could you help me out a bit with src/doc/man? Are the adoc files edited by hand and then everything else is generated?

[ehuss]

@ehuss could you help me out a bit with src/doc/man? Are the adoc files edited by hand and then everything else is generated?

Yes, the adoc files are written by hand. You need to install asciidoctor and run make in src/doc. I can write it and push to your branch if you want.

[alexcrichton]

Now with documentation!

[ehuss]

To add the man page to the website, it needs to be added to src/doc/src/SUMMARY.md. I'm not sure which category to put it in, maybe "manifest commands" or create a new category? Then add a markdown file that includes the html (see src/doc/src/commands/cargo-build.md as an example).

(I have to go, not sure what is wrong with ignore_files extracting a file, will take a look when I return.)

[ehuss]

Looks great! Is this ready to go? If so, r=me. It wasn't clear to me if there was more you wanted to do now.

[bors]

☔ The latest upstream changes (presumably #6865) made this pull request unmergeable. Please resolve the merge conflicts.

[alexcrichton]

Do you have thoughts on the breakage relative to cargo-vendor itself? I'm personally ok landing this and just saying "use cargo-vendor vendor ... if you're using the options, but we could also possibly go ahead and implement them in Cargo behind -Z or something like that.

[ehuss]

Just to be clear, is this the set of flags we are talking about?

-x, --explicit-version   Always include version in subdir name
--disallow-duplicates    Disallow two versions of one crate
--only-git-deps          Only vendor git dependencies, not crates.io dependencies
--relative-path          Use relative vendor path for .cargo/config
--no-merge-sources       Keep sources separate

I have no sense of how many people are using these flags, or how you decided which to leave out. Googling around, I don't see much. Like it looks like --explicit-version was once used by fuchsia, but it doesn't look like they use it anymore?

Absent any more information, I would lean towards what you said (use cargo-vendor as a fallback), and hopefully if someone is relying on one of these flags, they can file an issue. Otherwise, we could assume they are not needed anymore.

Perhaps we could just add the flags here with the hidden attribute, and check if they are used and raise an error informing the user why the flag is not supported anymore, and maybe ask them to file an issue here?

[alexcrichton]

Yeah it's just those flags, and I also don't have a great sense of how often they're used. I like the idea of hidden help flags that redirect users to file issues here, so I'll implement that.

[alexcrichton]

Alright, that makes me comfortable with this PR, so...

@rfcbot fcp merge

[rfcbot]

Team member @alexcrichton has proposed to merge this. The next step is review by the rest of the tagged team members:

• @Eh2406
• @alexcrichton
• @ehuss
• @joshtriplett
• @nrc
• @withoutboats

Concerns:

• why better to be built in? resolved by Import the cargo-vendor subcommand into Cargo #6869 (comment)

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[nrc]

Do you have a sense of who and how many people use cargo vendor in its current state? What is the motivation for moving it into Cargo rather than maintaining it as an 'official' plugin?

[nrc]

@rfcbot concern why better to be built in?

(see above comment)

[alexcrichton]

A good question! We talked about this briefly in the last cargo mtg but I'm realizing now I never actually wrote down the rationale, so now's a good a time as any!

I get the impression that usage of cargo vendor is quite common and is the defacto way to vendor dependencies (as opposed to cargo-local-registry). This is largely from the number of feature requests I've gotten on cargo-vendor over time and the support in maintaining it. Given that it's such an important part of our offline story as well it seemed like it was time to move it into Cargo proper to say that we support first-class vendoring rather than having to go through an external subcommand.

It's certainly possible to keep it as an external crate, but it comes with all the drawbacks of "oh now you have to cargo install it". To me it's just crossed the threshold of "useful enough to be bundled directly" sort of like cargo fix

[bors]

☔ The latest upstream changes (presumably #6880) made this pull request unmergeable. Please resolve the merge conflicts.

[LegNeato]

cargo-raze relies on -x when vendoring in: https://github.com/google/cargo-raze/blob/master/README.md

[cuviper]

FWIW, I also found cargo-vendor useful and important enough to include in Red Hat's rust-toolset offering -- so far it's the only thing we add beyond the rust "extended tools" group. I would be happy to drop the extra package. :)

[vi]

"oh now you have to cargo install it"

Can it be a rustup component? Or is rustup also met with some resistance?