Include source packages in manifest

[Diggsey]

This will allow rustup to install the rust source as an optional extension for a given toolchain.

[rust-highfive]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @brson (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

[Diggsey]

The source packages are a little different from other packages, in that they are not in the usual "installer" format (they don't have a components file describing how to install the package).

From rustup's POV, I plan on treating packages which don't contain a components file as "pure data" packages, which will just get copied to a suitably named subdirectory within the toolchain directory.

[alexcrichton]

Thanks for the PR! I suspect that @brson has many thoughts on this, so I'm somewhat hesitant to deploy a new format in the interim before he weighs in. I personally feel somewhat odd about using "*" as the "target" here, but so long as it works it's just an implementation detail so I'm not too worried.

[rillian]

Something like this is important I think to support the IDE integration tools which need copies of libcore and libstd for indexing. Having the installer put the source files in $prefix/src/ would make the sources easier to find.

[alexcrichton]

Ah yeah we definitely want to include them! This is just a question of how we modify our metadata to include info about where to download them from.

[brson]

Thanks for moving on this.

It looks to me like this is including the existing source tarball in the manifest, which is not in the rust-installer format. If we were to go that route then clients interpreting the manifest would have to know that tarball is special, and I prefer not to do that. So before we do this we need to modify the rust build system to produce a new source artifact that is built by rust-installer, so that rustup can treat it like any other installer.

The current source tarball is called "rustc-$version-src.tar.gz", and we'll want the build system to additionally produce a rust-installer tarball called "rustc-src-$version.tar.gz".

As to the installation location, it should probably be $prefix/lib/rustlib/src, and that will need to be configured when the rust-installer package is built in the rust build system.

The question of what to do with the target name is tricky. Fortunately rust-installer itself doesn't know anything about targets - it's just part of the package name. rustup though does care about targets and expects every component to have a name/target pair, so it will need to be extended to deal with components that don't have a target. I think using an asterisks here will be fine but I'm not sure of all the details it will affect.

[Diggsey]

I'd prefer to use rust-src or similar rather than rustc-src since this contains both the source for the standard library and rustc.

This shouldn't cause much of a problem for rustup: the modifications (if any are even needed) should be fairly self contained as it only uses the package targets to immediately lookup that package within the manifest.

I chose to use asterisks because it required the minimal number of changes, so made sense to do that until I could get some feedback, but modifying the manifest format also works.

[brson]

Oh, yes I prefer rust-src too. Typo.

[liuchong]

will it also set a "RUST_SRC_PATH" env var? racer and other apps are using this.

[retep998]

@liuchong That would probably be handled by a combination of rustup.rs and whatever editor tooling you're using. This PR is just allowing for source packages to be created on the buildbots.

[brson]

@Diggsey It looks to me like the only thing that needs to change is the naming scheme here. It's rust-src-$version-or-channel.tar.gz now.

[Diggsey]

Fixed up the name 😀

[brson]

I'm going to test this in the dev environment to make sure existing rustup deployments don't break on the new metadata. I suspect there are some tweaks needed still. Starting a build in dev now.

[brson]

The only thing that needs to change is that [pkg.rust-src.target.*] needs to output as [pkg.rust-src.target."*"].