Produce source package in rust-installer format #34366

Diggsey · Jun 19, 2016

See rust-lang-deprecated/rust-buildbot#102

There may be a better way to do this, wasn't sure how to clean-up the rust-src-image directory when it's used by multiple make rules.

rust-highfive · Jun 19, 2016

r? @brson

(rust_highfive has picked a reviewer for you, use r? to override)

alexcrichton · Jun 19, 2016

Thanks! Could you also update src/bootstrap/build/dist.rs with a rule for this as well?

Diggsey · Jun 19, 2016

The rustbuild system doesn't seem to produce a source tarball at all right now - should I make it produce the vanilla tarball in addition to the rust-installer package?

alexcrichton · Jun 20, 2016

Oh whoops! Eh it's fine to not do that as part of this PR if you'd prefer, but if you wanna throw it in I'm also fine with that!

brson · Jun 20, 2016

brson reviewed Jun 20, 2016

View changes

brson · Jun 20, 2016

mk/dist.mk

+	$(Q)$(S)src/rust-installer/gen-installer.sh \
+		--product-name=Rust \
+		--rel-manifest-dir=rustlib \
+		--success-message=Awesome-Source. \


brson · Jun 20, 2016

brson reviewed Jun 20, 2016

View changes

brson · Jun 20, 2016

mk/dist.mk

-$(PKG_TAR): $(PKG_FILES)
-	@$(call E, making dist dir)
-	$(Q)rm -Rf tmp/dist/$(PKG_NAME)
-	$(Q)mkdir -p tmp/dist/$(PKG_NAME)


Unfortunately I think we need to keep producing both formats. The old format is for people building rust, the new one is for people installing it for other purposes.

Oh, I think I see what you are doing to produce both.

Yep it should still produce the old .tar.gz exactly as before

brson · Jun 20, 2016

brson reviewed Jun 20, 2016

View changes

brson · Jun 20, 2016

mk/dist.mk

@@ -87,10 +88,11 @@ $(PKG_TAR): $(PKG_FILES)
         --exclude=*/llvm/test/*/*/*.ll \
         --exclude=*/llvm/test/*/*/*.td \
         --exclude=*/llvm/test/*/*/*.s \
-         -c $(UNROOTED_PKG_FILES) | tar -x -f - -C tmp/dist/$(PKG_NAME)
+         -c $(UNROOTED_PKG_FILES) | tar -x -f - -C tmp/dist/$(SRC_PKG_NAME)-image/lib/rust-lib/rust-src


rust-lib should be rustlib. It's the name of the directory we stuff arbitrary things into.

Yeah, I've already corrected that locally

brson · Jun 20, 2016

brson reviewed Jun 20, 2016

View changes

brson · Jun 20, 2016

mk/dist.mk

 	@$(call E, making $@)
-	$(Q)tar -czf $(PKG_TAR) -C tmp/dist $(PKG_NAME)
-	$(Q)rm -Rf tmp/dist/$(PKG_NAME)
+	$(Q)tar -czf $(PKG_TAR) -C tmp/dist/$(SRC_PKG_NAME)-image/lib/rust-lib rust-src --transform 's,^rust-src,$(PKG_NAME),S'


What does --transform 's,^rust-src,$(PKG_NAME),S' do?

Both source packages are built from the same image, and the directory name is rust-src, but tar uses the directory name as the root directory in the archive, whereas we used to name the root directory $(PKG_NAME), so the --transform option rewrites all the paths so that the resulting .tar.gz matches exactly the format we used before.

brson · Jun 20, 2016

cc #19535

brson · Jun 20, 2016

So after installation of the rust-src package the full Rust source will be at lib/rustlib/rust-src meaning the source to std will be at lib/rustlib/rust-src/src/libstd.

We've also discussed about the potential to need other source installations, but without concrete use cases. You might imagine that rust-src gets split into rust-src, rust-core-src, rustc-src etc. or we add a cargo-src. Should we consider making the scheme for storing the source expandable by putting this at e.g. lib/rust/lib/src/rust/?

cc @phildawes @alexreg @MarkSwanson this PR is deciding the scheme for storing the installed Rust source on disk. Once we deploy it it will be very hard to change.

cc brson/multirust#77

brson · Jun 20, 2016

It might also be worth considering the impact on side-by-side installation. Right now it's in theory vaguely possible to install multiple Rust's right on top of each other because of the disambiguated crate names, though in practice there are artifacts in rustlib that are not disambiguated. Putting a single copy of the source at a known location in rustlib means that only one can exist at a time. This is fine for rustup, but not fine for e.g. Linux distros.

I think I'm kinda fine just saying that every rustlib belongs to a single Rust installation, but it does put a nail in the coffin of any hope of treating rustlib as a more general 'global assembly cache'.

brson · Jun 21, 2016

cc @jauhien @vadimcn @sylvestre packaging people. This PR is trying to finalize the storage location for the Rust source code (#19535), for use by various tools, and I want to check what impact it might have on packagers. In rustup it'll be an optional package called rust-src, and tentatively installed to lib/rustlib/rust-src.

The concern I have regarding distro packaging is the impact on side-by-side installation. In particular, if SxS is achieved by just installing multiple revisions of Rust directly on top of each other then this src location doesn't work because multiple revisions will want to put the source there.

From previous conversations with @jauhien my impression was that at least Gentoo had a more sophisticated SxS installation scheme of their own. What I would like to do for simplicity is abandon any pretense that rustlib is shared by multiple versions of Rust. Let distros deal with SxS installation their own way and reexamine how it impacts us when the time comes.

Of course distros may not want to use our scheme for installing the source at all, but then they are going to have to deal with the integration with tools separately.

Diggsey · Jun 21, 2016

I've updated rustbuild to produce the two source distributions, and fixed the typo in rustlib.

durka · Jun 21, 2016

durka reviewed Jun 21, 2016

View changes

durka · Jun 21, 2016

src/bootstrap/build/dist.rs

@@ -287,6 +289,97 @@ pub fn std(build: &Build, compiler: &Compiler, target: &str) {
    t!(fs::remove_dir_all(&image));
 }

+/// Creates the `rust-std` installer component as compiled by `compiler` for the
+/// target `target`.


copy-pasted comment?

alexcrichton · Jun 21, 2016

I'd personally lean a bit towards rustlib/src/rust instead of rustlib/rust-src just to allow for Cargo one day (and perhaps other tools like rustfmt), but I don't think we'll go too far in the directon of rustlib/rust-core-src as it'd basically just be a space optimization.

MarkSwanson · Jun 21, 2016

I'm glad this is being worked on!
I like rustlib/src/rust.
I like making it easy to perhaps add the sources for cargo / rustfmt etc. Maybe llvm headers. clippy? etc.

brson · Jun 23, 2016

Let's go with rustlib/src/rust and keep pressing forward.

sylvestre · Jun 24, 2016

Sorry for the dumb question but what is SxS ?
Besides that, we don't ship the sources as a package in Debian. We are using the tag from github to get the tarball.

sorear · Jun 25, 2016

@sylvestre side-by-side installation, in other words having more than one version of rust (e.g. nightly and stable) installed and usable on the same machine at the same time

bors · Jul 2, 2016

☔️ The latest upstream changes (presumably #34611) made this pull request unmergeable. Please resolve the merge conflicts.

brson · Jul 8, 2016

Something to consider re the std-aware cargo RFC: in order to teach cargo how to build std, we're most likely going to end up having cargo look in the sysroot for the source - that is, the source we're packaging in this PR.

So far we've not made any particular commitments about what the 'shape' of that source will like, we're just dumping the source tree into rustlib/src/rust. ISTM though that Cargo might want a deterministic scheme for finding crates with a specific name, so that if it learns that e.g. collections is part of the crate tag, it knows to look in a specific directory for a Cargo.toml corresponding to the collections package.

ISTM there's also some overlap here with the idea of 'local crates.io mirrors' as well. @alexcrichton does it make sense for Cargo to treat part of the sysroot as a standard local crates.io mirror?

I guess I see a few options here:

we can go forward with this 'dump' of the raw source into the sysroot and expect tools to just kinda do there best (I imagine they would have a two stage lookup: first, use hardcoded knowledge of the official Rust source tree to find a crate; second walk the tree looking for Cargo.toml files)
we can instead plan a more structured layout for source in the sysroot where tools know exactly how to find the source to a specific crate
we can go ahead with the current patch but plan on adding the additional structure later.
a combination of the above, but structuring the source in the sysroot to fit Cargo's hypothetical 'local registry' layout.

For the sake of expediency I think I'm inclined to just keep going forward with this patch and force tools like Cargo to do their bust to locate the crates in the source tree.

cc @Ericson2314

brson · Jul 8, 2016

@Diggsey what are the prospects for whipping this patch back into shape?

Diggsey · Jul 8, 2016

Just need to fix the merge conflicts then in that case.

eternaleye · Jul 8, 2016

@brson: Personally, I'd prefer option 3 - roll this out as is, but plan on adding structure.

In the end, that results in the same two-phase lookup being written into tools, but rather than privileging rustc (and making every other compiler implementation take a perf hit unless they mimic a historical accident, and one that may change in the future if rustc's repo changes anything), it makes the fallback less and less frequently-hit over time.

It's just as expedient for Rust (no extra work before rollout), more expedient for Cargo (implement only the "look for Cargo.toml" path now), and doesn't privilege rustc over future implementations (at least one is already being written as mrustc.)

alexcrichton · Jul 9, 2016

@brson this is basically already in the shape of a "path source" that Cargo expects today (or a git repository), so it should interoperate regardless. If we want to work with a local registry or vendored directory then we'd probably want more structure, but as you say I'm fine adding that later.

Diggsey · Jul 9, 2016

Fixed conflicts.

Diggsey · Aug 9, 2016

Failure looks to be spurious, and was caused by remove_dir_all being broken on windows (#29497)

brson · Aug 10, 2016

@bors retry

bors · Aug 10, 2016

⌛️ Testing commit 777bd6f with merge 0eefd73...

bors · Aug 10, 2016

💔 Test failed - auto-mac-64-opt-rustbuild

Diggsey · Aug 11, 2016

Ugh... That should fix it for mac. 🍎 🔫

brson · Aug 11, 2016

@bors r+

bors · Aug 11, 2016

📌 Commit 511c0d1 has been approved by brson

bors · Aug 11, 2016

⌛️ Testing commit 511c0d1 with merge 7f6705d...

bors · Aug 11, 2016

💔 Test failed - auto-mac-64-opt-rustbuild

alexcrichton · Aug 11, 2016

@bors: retry

bors · Aug 12, 2016

⌛️ Testing commit 511c0d1 with merge 1447527...

bors · Aug 12, 2016

💔 Test failed - auto-mac-64-opt-rustbuild

Diggsey · Aug 12, 2016

I think this is caused by the addition of the metadata section and package checksums to the Cargo.lock format in new versions of cargo, but I have no idea why other PRs aren't affected, or why a different version of cargo would be being used.

Mark-Simulacrum · Aug 12, 2016

Other PRs wouldn't be affected if they didn't change Cargo.toml as far as I know; I think this is true of most PRs.

alexcrichton · Aug 12, 2016

Lots of Cargo.lock exists that shouldn't in those logs, for example libpanic_unwind/Cargo.lock shouldn't ever exist. I wonder how they're being created?

The Cargo in use to generate these is pinned, it's downloaded as part of the bootstrap process

Diggsey · Aug 13, 2016

@alexcrichton Those files exist in the repository...

Diggsey · Aug 13, 2016

This should fix the tidy check.

There were two problems:

This specific check did not work on windows. msys's git diff-index ignores filenames with backslashes in, so the files would always show as unchanged. I fixed this by replacing backslashes with forward slashes before passing it to git.
git diff-index produces false positives after running make dist. I still haven't figured out why this happens, but using git diff instead fixes the problem.

alexcrichton · Aug 14, 2016

@bors: r=brson b3908d0

bors · Aug 14, 2016

⌛️ Testing commit b3908d0 with merge 92ae4ce...

bors · Aug 14, 2016

Diggsey · Aug 14, 2016

🎆

rust-highfive assigned brson Jun 19, 2016

Diggsey force-pushed the Diggsey:rust-src-pkg branch from fc5b57c to bb938d1 Jun 23, 2016

Diggsey force-pushed the Diggsey:rust-src-pkg branch from bb938d1 to 81673eb Jun 25, 2016

Diggsey force-pushed the Diggsey:rust-src-pkg branch from 81673eb to 21ccad9 Jul 9, 2016

Diggsey force-pushed the Diggsey:rust-src-pkg branch from 777bd6f to 511c0d1 Aug 11, 2016

Produce source package in rust-installer format in addition to vanill…
…a tarball Copy source files from rust code Add missing wildcard Remove unused function Remove use of tar --transform

7341d68

Fix make-tidy lock file checks

Loading status checks…

b3908d0

Diggsey force-pushed the Diggsey:rust-src-pkg branch from 511c0d1 to b3908d0 Aug 13, 2016

bors merged commit b3908d0 into rust-lang:master Aug 14, 2016
2 checks passed

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

homu Test successful
Details

rust-lang/rust

Join GitHub today

Produce source package in rust-installer format #34366

Conversation

Diggsey commented Jun 19, 2016

rust-highfive assigned brson Jun 19, 2016

This comment has been minimized.

rust-highfive commented Jun 19, 2016

This comment has been minimized.

alexcrichton commented Jun 19, 2016

This comment has been minimized.

Diggsey commented Jun 19, 2016

This comment has been minimized.

alexcrichton commented Jun 20, 2016

brson reviewed Jun 20, 2016 View changes

This comment has been minimized.

brson reviewed Jun 20, 2016 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

brson reviewed Jun 20, 2016 View changes

This comment has been minimized.

This comment has been minimized.

brson reviewed Jun 20, 2016 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

brson commented Jun 20, 2016

This comment has been minimized.

brson commented Jun 20, 2016

This comment has been minimized.

brson commented Jun 20, 2016

This comment has been minimized.

brson commented Jun 21, 2016

This comment has been minimized.

Diggsey commented Jun 21, 2016

durka reviewed Jun 21, 2016 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

alexcrichton commented Jun 21, 2016

This comment has been minimized.

MarkSwanson commented Jun 21, 2016

This comment has been minimized.

brson commented Jun 23, 2016

Diggsey force-pushed the Diggsey:rust-src-pkg branch from fc5b57c to bb938d1 Jun 23, 2016

This comment has been minimized.

sylvestre commented Jun 24, 2016

Diggsey force-pushed the Diggsey:rust-src-pkg branch from bb938d1 to 81673eb Jun 25, 2016

This comment has been minimized.

sorear commented Jun 25, 2016

This comment has been minimized.

bors commented Jul 2, 2016

This comment has been minimized.

brson commented Jul 8, 2016

This comment has been minimized.

brson commented Jul 8, 2016

This comment has been minimized.

Diggsey commented Jul 8, 2016

This comment has been minimized.

eternaleye commented Jul 8, 2016

This comment has been minimized.

alexcrichton commented Jul 9, 2016

Diggsey force-pushed the Diggsey:rust-src-pkg branch from 81673eb to 21ccad9 Jul 9, 2016

This comment has been minimized.

Diggsey commented Jul 9, 2016

This comment has been minimized.

Diggsey commented Aug 9, 2016

This comment has been minimized.

brson commented Aug 10, 2016

This comment has been minimized.

bors commented Aug 10, 2016

bors added a commit that referenced this pull request Aug 10, 2016

This comment has been minimized.

bors commented Aug 10, 2016

Diggsey force-pushed the Diggsey:rust-src-pkg branch from 777bd6f to 511c0d1 Aug 11, 2016

This comment has been minimized.

Diggsey commented Aug 11, 2016

This comment has been minimized.

brson commented Aug 11, 2016

brson reviewed Jun 20, 2016

View changes

brson reviewed Jun 20, 2016

View changes

brson reviewed Jun 20, 2016

View changes

brson reviewed Jun 20, 2016

View changes

durka reviewed Jun 21, 2016

View changes

Diggsey force-pushed the Diggsey:rust-src-pkg branch from `fc5b57c` to `bb938d1` Jun 23, 2016

Diggsey force-pushed the Diggsey:rust-src-pkg branch from `bb938d1` to `81673eb` Jun 25, 2016

Diggsey force-pushed the Diggsey:rust-src-pkg branch from `81673eb` to `21ccad9` Jul 9, 2016

Diggsey force-pushed the Diggsey:rust-src-pkg branch from `777bd6f` to `511c0d1` Aug 11, 2016

Diggsey force-pushed the Diggsey:rust-src-pkg branch from `511c0d1` to `b3908d0` Aug 13, 2016

bors merged commit `b3908d0` into rust-lang:master Aug 14, 2016
2 checks passed