Streamline `Compiler` #64016

nnethercote · 2019-08-30T07:38:08Z

A few commits to clean up Compiler.

bors · 2019-08-30T13:58:45Z

☔ The latest upstream changes (presumably #63827) made this pull request unmergeable. Please resolve the merge conflicts.

Zoxc · 2019-08-30T14:44:30Z

Some of the changes here seems questionable (the shuffling of code and the removal of compile). This also conflicts with #59904.

nnethercote · 2019-08-30T21:25:35Z

This is where I express some frustration.

Your feedback is not actionable. I have spent time trying to improve the quality of the code. Please be more specific in your criticisms, so that I (a) actually know what they are, and (b) can respond to them and/or act on them.
You have said this conflicts with Remove queries from rustc_interface #59904. That is one of a chain of 5 PRs. None of them have any explanation of what they are doing beyond the PR title. They all date from March, two of them (Remove queries from rustc_interface #59904 and Turn parsing into a query #59338) are closed, and the other two are tagged with S-blocked or S-waiting-on-team. It is difficult for me to understand the intention and status of these PRs in their current state.
My motivation for looking at this code was that I want to try moving metadata generation earlier, to see if it improves the effectiveness of pipelining. This requires understanding the main passes of the compiler, which turns out to be far more complex than I expected due to the implicit ordering that occurs in this query-based design. My first attempt had no effect because I didn't realize that passes could be called multiple times, with the latter calls effectively being no-ops. I spent an afternoon diagramming the passes and their dependencies in order to understand how it works. (The diagram is below.) The changes in this PR minimize and simplify dependencies, merge two unnecessarily-separated passes, and remove an inconsistent function. I think they are worthwhile, and I would appreciate a more respectful engagement with them.

nnethercote · 2019-08-31T00:31:37Z

Here is an updated picture showing things after my changes. The graph is neater partly because I did a better job drawing it, but also partly because it's simpler: it has three fewer boxes (passes::register_plugins, compile, and ongoing_codegen & link have been combined into codegen_and_link) and fewer arrows.

It's a false dependency. The result isn't used and there are no relevant side-effects.

nnethercote · 2019-09-01T23:40:38Z

#63827 landed over the weekend, which removed one of the Compiler::compile() calls. As a result, the last patch is now more compelling.

nnethercote · 2019-09-03T04:03:05Z

The S-waiting-on-author tag is incorrect, but I don't know how to remove it.

nnethercote · 2019-09-05T00:17:51Z

r? @oli-obk

Because they reviewed #56732.

oli-obk · 2019-09-06T12:15:34Z

src/librustc_driver/lib.rs

-
-        // Drop GlobalCtxt after starting codegen to free memory
-        mem::drop(compiler.global_ctxt()?.take());
+        compiler.codegen_and_link()?;


having reviewed @Zoxc's PRs it seems to me like only this commit is actually problematic for the querification PRs (like those PRs would have to revert this commit to be doable). As far as I can tell (both from the code and your graphs) it's not that much additional complexity to keep around either.

What do you think?

Thank you for the feedback. Are you referring to #59404? If so:

If there's a good reason to keep the ongoing_codegen/link split then I am fine to remove this commit.

Is there a good reason? Does Make ongoing_codegen a query #59404 and its predecessors use the existing passes because they are there, rather than because they are necessary?

If there's a good reason to keep the ongoing_codegen/link split then I am fine to remove this commit.

The main reason to keep this separate because ongoing_codegen needs the GlobalCtxt to be around, while we want to free GlobalCtxt before linking. When we remove queries from rustc_interface this mean this cannot be a combined operation.

Another reason is that it's reasonable for front-ends to do only code generation, but no linking, although that is not yet possible.

I don't understand the point about GlobalCtxt. My patch preserves the "free GlobalCtxt before linking" behaviour.

As for the second point, I would argue that YAGNI applies.

The problem isn't that your PR doesn't keep the early drop, it's the way the query PR works, where the GlobalCtxt lives as long as a closure and is automatically dropped at the end of it. So you can't call the combined function inside the closure and you can't call it outside the closure, thus needing to split it up again.

Zoxc · 2019-09-07T02:54:46Z

src/librustc_interface/passes.rs

@@ -261,9 +260,6 @@ pub fn register_plugins<'a>(
        });
    }

-    // If necessary, compute the dependency graph (in the background).
-    compiler.dep_graph_future().ok();
-


Moving the dep graph loading to a later point is a performance regression. This is currently done as early as possible, although I'd like for it be done immediately when creating the compiler context, but that requires some changes as currently we need to do this after parsing to pick up the crate name.

The performance run shows that these five commits provide a slightly performance improvement.

Zoxc · 2019-09-07T03:33:06Z

src/librustc_interface/queries.rs

            ).map_err(|_| ErrorReported)?;

            Ok(())
        })
    }
-
-    pub fn compile(&self) -> Result<()> {


This is meant as an simple utility function and an example for users of rustc_interface of something that mirrors a regular compilation session without all the complexities of the full rustc driver (in rustc_driver).

If an example is needed, why not provide it in documentation? Don't leave unnecessary code in the product.

Do we have CI testable example code snippets in the rustc-guide?

Or maybe we could put such a snippet into the rustdoc comments that we process for rustc's generated documentation?

If either of the above is true, then I could go along with @nnethercote 's reasoning here, though then I would want that migration to happen either before this PR, or as part of this PR.

(Otherwise, I'm with @Zoxc; Its hard enough for people to get started using the compilier as a library or hacking on rustc itself; simple examples of how to use things like rustc_interface are in their own way part of our product...)

nnethercote · 2019-09-07T21:01:21Z

@bors try

bors · 2019-09-07T21:01:34Z

⌛ Trying commit 92a0cbd with merge 468729d...

@Zoxc

Streamline `Compiler` A few commits to clean up `Compiler`. r? @Zoxc

bors · 2019-09-07T23:41:33Z

☀️ Try build successful - checks-azure
Build commit: 468729d

nnethercote · 2019-09-08T00:07:14Z

@rust-timer build 468729d

rust-timer · 2019-09-08T00:07:15Z

Success: Queued 468729d with parent ef54f57, comparison URL.

rust-timer · 2019-09-08T03:45:11Z

Finished benchmarking try commit 468729d, comparison URL.

`Compiler::register_plugins()` calls `passes::register_plugins()`, which calls `Compiler::dep_graph_future()`. This is the only way in which a `passes` function calls a `Compiler` function. This commit moves the `dep_graph_future()` call out of `passes::register_plugins()` and into `Compiler::register_plugins()`, which is a more sensible spot for it. This will delay the loading of the dep graph slightly -- from the middle of plugin registration to the end of plugin registration -- but plugin registration is fast enough (especially compared to expansion) that the impact should be neglible.

nnethercote · 2019-09-11T02:14:21Z

I removed the commit that merges ongoing_codegen with link. I think the other four commits are still worth landing.

`Compiler::compile()` is different to all the other `Compiler` methods because it lacks a `Queries` entry. It only has one call site, which is in a test that doesn't need its specific characteristics. This patch replaces that call with a call to `Compile::link()`, which is similar enough for the test's purposes. It also notes that the method is an illustrative example of how `Compiler` can be used.

nnethercote · 2019-09-18T20:58:21Z

I have changed the fourth commit to retain Compiler::compile, and added a comment explaining that it's a useful example.

The four commits as they now stand.

Don't call self.parse() in Compiler::crate_name() unless necessary. Trivial, should be uncontroversial.
Remove lower_to_hir() call from prepare_output(). Very simple, should be uncontroversial.
Move call site of dep_graph_future(). A clear simplification. The perf results above should allay concerns about perf regressions.
Add a comment to Compiler::compile(). Trivial, should be uncontroversial.

@oli-obk: can you re-review? Thanks.

oli-obk · 2019-09-24T20:50:23Z

@bors r+

bors · 2019-09-24T20:50:24Z

📌 Commit 2521189 has been approved by oli-obk

@Zoxc

…i-obk Streamline `Compiler` A few commits to clean up `Compiler`. r? @Zoxc

@ghost

Rollup of 16 pull requests Successful merges: - #63356 (Issue#63183: Add fs::read_dir() and ReadDir warning about iterator order + example) - #63934 (Fix coherence checking for impl trait in type aliases) - #64016 (Streamline `Compiler`) - #64296 (Document the unstable iter_order_by library feature) - #64443 (rustdoc: general cleanup) - #64622 (Add a cycle detector for generic `Graph`s and `mir::Body`s) - #64689 (Refactor macro by example) - #64698 (Recover on `const X = 42;` and infer type + Error Stash API) - #64702 (Remove unused dependencies) - #64717 (update mem::discriminant test to use assert_eq and assert_ne over comparison operators) - #64720 ( remove rtp.rs, and move rtpSpawn and RTP_ID_ERROR to libc) - #64721 (Fixed issue from #64447) - #64725 (fix one typo) - #64737 (fix several issues in String docs) - #64742 (relnotes: make compatibility section more sterile and fix rustc version) - #64748 (Fix #64744. Account for the Zero sub-pattern case.) Failed merges: r? @ghost

rust-highfive assigned Zoxc Aug 30, 2019

bors added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Aug 30, 2019

nnethercote changed the title ~~Streamling Compiler~~ Streamline Compiler Aug 30, 2019

nnethercote added 2 commits September 2, 2019 08:46

Don't call self.parse() in Compiler::crate_name() unless necessary.

d61605c

Remove lower_to_hir() call from prepare_output().

cd0c21b

It's a false dependency. The result isn't used and there are no relevant side-effects.

nnethercote force-pushed the Compiler-fiddling branch from c1c386e to 392b91f Compare September 1, 2019 23:39

nnethercote force-pushed the Compiler-fiddling branch from 392b91f to 92a0cbd Compare September 1, 2019 23:41

nnethercote mentioned this pull request Sep 3, 2019

Move metadata generation before analysis #64112

Closed

This comment has been minimized.

Sign in to view

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Sep 3, 2019

rust-highfive assigned oli-obk and unassigned Zoxc Sep 5, 2019

oli-obk reviewed Sep 6, 2019

View reviewed changes

Zoxc reviewed Sep 7, 2019

View reviewed changes

bors added a commit that referenced this pull request Sep 7, 2019

Auto merge of #64016 - nnethercote:Compiler-fiddling, r=<try>

468729d

Streamline `Compiler` A few commits to clean up `Compiler`. r? @Zoxc

nnethercote force-pushed the Compiler-fiddling branch from 92a0cbd to 443e6d2 Compare September 11, 2019 02:13

nnethercote force-pushed the Compiler-fiddling branch from 443e6d2 to bdf5b54 Compare September 11, 2019 02:15

nnethercote force-pushed the Compiler-fiddling branch from bdf5b54 to 8e53227 Compare September 18, 2019 20:54

nnethercote force-pushed the Compiler-fiddling branch from 8e53227 to 2521189 Compare September 18, 2019 20:57

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 24, 2019

Centril added a commit to Centril/rust that referenced this pull request Sep 24, 2019

Rollup merge of rust-lang#64016 - nnethercote:Compiler-fiddling, r=ol…

ad0b78d

…i-obk Streamline `Compiler` A few commits to clean up `Compiler`. r? @Zoxc

Centril mentioned this pull request Sep 24, 2019

Rollup of 16 pull requests #64751

Merged

bors merged commit 2521189 into rust-lang:master Sep 25, 2019

nnethercote deleted the Compiler-fiddling branch September 25, 2019 02:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Streamline `Compiler` #64016

Streamline `Compiler` #64016

nnethercote commented Aug 30, 2019

bors commented Aug 30, 2019

Zoxc commented Aug 30, 2019

nnethercote commented Aug 30, 2019

nnethercote commented Aug 31, 2019

nnethercote commented Sep 1, 2019

nnethercote commented Sep 3, 2019

This comment has been minimized.

nnethercote commented Sep 5, 2019

oli-obk Sep 6, 2019

nnethercote Sep 6, 2019

Zoxc Sep 7, 2019

nnethercote Sep 8, 2019

oli-obk Sep 8, 2019

Zoxc Sep 7, 2019

nnethercote Sep 8, 2019

Zoxc Sep 7, 2019

nnethercote Sep 8, 2019

pnkfelix Sep 17, 2019

nnethercote commented Sep 7, 2019

bors commented Sep 7, 2019

bors commented Sep 7, 2019

nnethercote commented Sep 8, 2019

rust-timer commented Sep 8, 2019

rust-timer commented Sep 8, 2019

nnethercote commented Sep 11, 2019

nnethercote commented Sep 18, 2019

oli-obk commented Sep 24, 2019

bors commented Sep 24, 2019

Streamline Compiler #64016

Streamline Compiler #64016

Conversation

nnethercote commented Aug 30, 2019

bors commented Aug 30, 2019

Zoxc commented Aug 30, 2019

nnethercote commented Aug 30, 2019

nnethercote commented Aug 31, 2019

nnethercote commented Sep 1, 2019

nnethercote commented Sep 3, 2019

This comment has been minimized.

nnethercote commented Sep 5, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nnethercote commented Sep 7, 2019

bors commented Sep 7, 2019

bors commented Sep 7, 2019

nnethercote commented Sep 8, 2019

rust-timer commented Sep 8, 2019

rust-timer commented Sep 8, 2019

nnethercote commented Sep 11, 2019

nnethercote commented Sep 18, 2019

oli-obk commented Sep 24, 2019

bors commented Sep 24, 2019

Streamline `Compiler` #64016

Streamline `Compiler` #64016