Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Query Parallelization Tracking Issue #48685

Open
nikomatsakis opened this issue Mar 2, 2018 · 9 comments
Open

Query Parallelization Tracking Issue #48685

nikomatsakis opened this issue Mar 2, 2018 · 9 comments

Comments

@nikomatsakis
Copy link
Contributor

@nikomatsakis nikomatsakis commented Mar 2, 2018

This issue is a sub-issue of #48547: it tracks the in-progress effort to parallelize rustc across queries. This work is being spearheaded by @Zoxc.

Goals

Allow rustc to execute queries in parallel with one another. Enable the use of rayon or other tools for intra-query parallelization as well. See this internals thread for more information.

Overview of the plan

  • Make all types in the compiler send and sync by landing @Zoxc's existing work
  • Set up CI so this gets some testing (#48607)
  • Implement a thread pool that is integrated with Cargo's jobserver (#56946)
  • Build list of all sources of shared state in the compiler by auditing PR history #63643
  • Audit each source of shared state, leading to either refactoring to reduce use of shared state, or persistent documentation covering invariants, atomicity, and lock orderings

Pending refactorings

  • The TransitiveRelation doesn't really need to use a RefCell. In the future, it won't even be shared, but regardless the caching scheme could be reworked to avoid RefCell.
  • ParseSess.included_mod_stack might be problematic
  • Ensure that the err_count() API is not used to discover if errors happen in parallel code (cc #49737)
  • Find a way to order error messages deterministically so that if a query depends on another query, its error messages appear after the other query (cc #49737)
  • Make mk_attr_id use a scoped thread local or make it part of ParseSess
  • Refactor GlobalCtxt::layout_depth so it does not need global mutable state (#49735)
  • See if GlobalCtxt.rcache, OnDiskCache.file_index_to_file and OnDiskCache.synthetic_expansion_infos are faster as thread-locals
  • Review usages of Session.lint_store and Session.buffered_lints
  • Fix #50507, as it may cause issues with parallel rustc
  • Find a way to deal with marking attributes as used
  • Ensure Rayon executes all remaining work when we panic inside the thread pool

Completed refactorings

  • Remove HIR inlining (#49690)
  • Make GlobalCtxt implement Sync
  • Use scoped thread locals instead of globals, #46193
  • Misc fixes required to use a rayon thread pool, #46564
  • Ensure metadata loaded by LlvmMetadataLoader does not get freed on the wrong thread
  • CurrentDepGraph.task_stack has to be made a QueryJob field. (implemented in #49732)
  • Check that optimization fuel is not used with parallel queries
  • Review libproc_macro for issues, Find out which types should be Send, Sync. Deal with Deref impls for Symbol. (#49219)
  • Run CI with cfg(parallel_queries)
  • Refactor away CStore::next_crate_num
  • Make FileMap.lines, FileMap.multibyte_chars, and FileMap.non_narrow_chars immutable (implemented in #50997)
  • When executing queries, instead of keeping around a lock to the query map, create a job object immediately instead (#50102)
  • Ensure all diagnostics are emitted before returning from DepGraph.try_mark_green
  • Refactor DepGraphData.previous_work_products so it becomes immutable (#50501) (implemented in #50524)
  • Remove DepGraphData.work_products by threaded the value through save_trans_partition() -> copy_module_artifacts_into_incr_comp_cache() -> OngoingCrateTranslation::join() (#50500) (implemented in #50885
@nikomatsakis
Copy link
Contributor Author

@nikomatsakis nikomatsakis commented Mar 2, 2018

@michaelwoerister
Copy link
Contributor

@michaelwoerister michaelwoerister commented May 7, 2018

I'm trying to identify the blocking issues for a minimal but correct version of parallel queries. Ideally we would solve all of them over the next few weeks :)

So far I've found:

  • MIRI const evaluation is not really thread-safe yet and needs at least a proper review for possible race conditions.
  • SelectionCache and EvaluationCache are not thread-safe yet. A PR is open (#49834) but needs review by @nikomatsakis.
  • Rayon needs some enhancements (#50235 (comment)) that have not been reviewed yet (@nikomatsakis or @cuviper).
  • Incremental compilation's OnDiskCache could use another review for race conditions.
  • GlobalCtxt is not Sync yet, which is blocked on a few of the above.
  • We actually need to use the parallel abstractions that @Zoxc introduced.

There are a few things that also need to be solved medium term, but they are not blocking yet:

  • Error message handling must be deterministic enough for tests to work (or tests need to handle indeterminism)
  • Various things could be made thread-local or immutable instead of globally shared.
  • Various corner cases (such as optimization fuel) may not work correctly yet.

@Zoxc, @nikomatsakis, @oli-obk, @eddyb: I'm sure this list does not cover all blockers. Let me know what I forgot :)

@Zoxc
Copy link
Contributor

@Zoxc Zoxc commented May 7, 2018

@michaelwoerister We also have to make queries work on a Rayon thread-pool

@michaelwoerister
Copy link
Contributor

@michaelwoerister michaelwoerister commented May 7, 2018

We also have to make queries work on a Rayon thread-pool

What exactly does this involve?

@Zoxc
Copy link
Contributor

@Zoxc Zoxc commented May 7, 2018

@michaelwoerister We have to make waiting on already executing queries block and also detect and recover from query cycles

@michaelwoerister
Copy link
Contributor

@michaelwoerister michaelwoerister commented May 7, 2018

Ah, right. The blocking should be easy enough. I have not thought in detail about cycle detection and recovery though.

@michaelwoerister
Copy link
Contributor

@michaelwoerister michaelwoerister commented Jul 4, 2018

We have to make waiting on already executing queries block and also detect and recover from query cycles

Both of these should be implemented by #50699, right?

@Zoxc
Copy link
Contributor

@Zoxc Zoxc commented Jul 4, 2018

@ghost
Copy link

@ghost ghost commented Jan 3, 2019

A rustc HEAD compiled with experimental-parallel-queries = true causes rust-clippy HEAD compilation to fail as seen here: rust-lang/rust-clippy#3613

bors added a commit that referenced this issue Mar 26, 2019
ci: Conditionally build parallel compiler on `try`

This commit configures Travis/AppVeyor to conditionally compile parallel
compilers on `@bors: try`. This is an experiment currently to see how
this plays out, but the intention is that if the commit message contains
the term "parallel-compiler" then when `@bors: try` is issued it will
perform differently than the try branch does today, building three
compilers: Linux, macOS, and Windows. We currently have no `try`
builders for macOS or Windows due to typical capacity issues, so it's
intended that this is only very sparingly used from time to time when
necessary.

[parallel-compiler]

cc #48685, tracking issue for parallel compilation
bors added a commit that referenced this issue Apr 1, 2019
ci: Conditionally build parallel compiler on `try`

This commit configures Travis/AppVeyor to conditionally compile parallel
compilers on `@bors: try`. This is an experiment currently to see how
this plays out, but the intention is that if the commit message contains
the term "parallel\-compiler" then when `@bors: try` is issued it will
perform differently than the try branch does today, building three
compilers: Linux, macOS, and Windows. We currently have no `try`
builders for macOS or Windows due to typical capacity issues, so it's
intended that this is only very sparingly used from time to time when
necessary.

cc #48685, tracking issue for parallel compilation
bors added a commit that referenced this issue Apr 1, 2019
ci: Conditionally build parallel compiler on `try`

This commit configures Travis/AppVeyor to conditionally compile parallel
compilers on `@bors: try`. This is an experiment currently to see how
this plays out, but the intention is that if the commit message contains
the term "parallel\-compiler" then when `@bors: try` is issued it will
perform differently than the try branch does today, building three
compilers: Linux, macOS, and Windows. We currently have no `try`
builders for macOS or Windows due to typical capacity issues, so it's
intended that this is only very sparingly used from time to time when
necessary.

cc #48685, tracking issue for parallel compilation
bors added a commit that referenced this issue Apr 1, 2019
ci: Conditionally build parallel compiler on `try`

This commit configures Travis/AppVeyor to conditionally compile parallel
compilers on `@bors: try`. This is an experiment currently to see how
this plays out, but the intention is that if the commit message contains
the term "parallel\-compiler" then when `@bors: try` is issued it will
perform differently than the try branch does today, building three
compilers: Linux, macOS, and Windows. We currently have no `try`
builders for macOS or Windows due to typical capacity issues, so it's
intended that this is only very sparingly used from time to time when
necessary.

cc #48685, tracking issue for parallel compilation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants