Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move metadata generation before analysis #64112

wants to merge 6 commits into from
Changes from 1 commit
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.


Just for now


[DRAFT] Move metadata generation before analysis.

The goal of this commit is to improve the effectiveness of pipelining,
because metadata generation must complete before dependent crates can
begin compiling.

Unfortunately it's not much of a win. The biggest speed-up I have seen
is 1.07x, vs 1.79x for the original pipelining implementation. And it
slightly slows down the common case when an error occurs during
analysis, because metadata generation now precedes analysis.

Currently three tests fail with the change, due to bounds violations.
These tests are currently `git rm`'d, so that subsequent tests can run.
  • Loading branch information...
nnethercote committed Sep 2, 2019
commit c5f5e5b5f655d20f1c3a0f3c19ae13ec19d4bb39
@@ -81,14 +81,16 @@ pub enum Linkage {

pub fn calculate(tcx: TyCtxt<'_>) {
pub fn calculate(tcx: TyCtxt<'_>, abort_if_errors: bool) {
let sess = &tcx.sess;
let fmts = sess.crate_types.borrow().iter().map(|&ty| {
let linkage = calculate_type(tcx, ty);
verify_ok(tcx, &linkage);
(ty, linkage)
}).collect::<FxHashMap<_, _>>();
if abort_if_errors {

@@ -347,6 +347,11 @@ pub fn run_compiler(
return sess.compile_status();

// We want metadata generation to be as early as possible, i.e. before
// the analysis phases, to maximize pipelining opportunities. But wait
// until after the analysis phases to abort if there are errors.
compiler.do_metadata(/* abort_if_errors */ false)?;

if sess.opts.debugging_opts.save_analysis {
let expanded_crate = &compiler.expansion()?.peek().0;
let crate_name = compiler.crate_name()?.peek().clone();
@@ -375,6 +380,8 @@ pub fn run_compiler(

compiler.global_ctxt()?.peek_mut().enter(|tcx| tcx.analysis(LOCAL_CRATE))?;


if callbacks.after_analysis(compiler) == Compilation::Stop {
return sess.compile_status();
@@ -1077,27 +1077,36 @@ fn encode_and_write_metadata(
(metadata, need_metadata_module)

/// Do metadata generation.
pub fn do_metadata<'tcx>(
tcx: TyCtxt<'tcx>,
outputs: &OutputFilenames,
abort_if_errors: bool,
) -> (middle::cstore::EncodedMetadata, bool) {
time(tcx.sess, "resolving dependency formats", || {
middle::dependency_format::calculate(tcx, abort_if_errors)

time(tcx.sess, "metadata encoding and writing", || {
encode_and_write_metadata(tcx, outputs)

/// Runs the codegen backend, after which the AST and analysis can
/// be discarded.
pub fn start_codegen<'tcx>(
codegen_backend: &dyn CodegenBackend,
tcx: TyCtxt<'tcx>,
rx: mpsc::Receiver<Box<dyn Any + Send>>,
metadata: middle::cstore::EncodedMetadata,
need_metadata_module: bool,
outputs: &OutputFilenames,
) -> Box<dyn Any> {
if log_enabled!(::log::Level::Info) {

time(tcx.sess, "resolving dependency formats", || {

let (metadata, need_metadata_module) = time(tcx.sess, "metadata encoding and writing", || {
encode_and_write_metadata(tcx, outputs)

tcx.sess.profiler(|p| p.start_activity("codegen crate"));
let codegen = time(tcx.sess, "codegen", move || {
codegen_backend.codegen_crate(tcx, metadata, need_metadata_module, rx)
@@ -6,6 +6,7 @@ use rustc::session::config::OutputFilenames;
use rustc::util::common::{time, ErrorReported};
use rustc::hir;
use rustc::hir::def_id::LOCAL_CRATE;
use rustc::middle::cstore;
use rustc::ty::steal::Steal;
use rustc::dep_graph::DepGraph;
use std::any::Any;
@@ -83,6 +84,7 @@ pub(crate) struct Queries {
codegen_channel: Query<(Steal<mpsc::Sender<Box<dyn Any + Send>>>,
Steal<mpsc::Receiver<Box<dyn Any + Send>>>)>,
global_ctxt: Query<BoxedGlobalCtxt>,
do_metadata: Query<(cstore::EncodedMetadata, bool)>,
codegen_and_link: Query<()>,

@@ -237,10 +239,23 @@ impl Compiler {

pub fn do_metadata(&self, abort_if_errors: bool)
-> Result<&Query<(cstore::EncodedMetadata, bool)>> {
self.queries.do_metadata.compute(|| {
let outputs = self.prepare_outputs()?;
Ok(self.global_ctxt()?.peek_mut().enter(|tcx| {
passes::do_metadata(tcx, &*outputs.peek(), abort_if_errors)

pub fn codegen_and_link(&self) -> Result<&Query<()>> {
self.queries.codegen_and_link.compute(|| {
let rx = self.codegen_channel()?.peek().1.steal();
let outputs = self.prepare_outputs()?;
let (metadata, need_metadata_module) =
self.do_metadata(/* abort_if_errors */ true)?.take();

let ongoing_codegen = self.global_ctxt()?.peek_mut().enter(|tcx| {

@@ -251,6 +266,8 @@ impl Compiler {

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.