Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Short circuit Send and Sync impls for TokenTree #60967

Merged
merged 1 commit into from May 27, 2019

Conversation

Projects
None yet
9 participants
@Zoxc
Copy link
Contributor

commented May 19, 2019

Workaround to make the parallel compiler build after #60444.

r? @nikomatsakis

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented May 19, 2019

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:2a7cc96e:start=1558293500505906466,finish=1558293501296013601,duration=790107135
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
$ export AWS_ACCESS_KEY_ID=AKIA46X5W6CZEJZ6XT55
---
[00:07:29]    Compiling arena v0.0.0 (/checkout/src/libarena)
[00:07:41]    Compiling rustc_macros v0.1.0 (/checkout/src/librustc_macros)
[00:07:52]    Compiling syntax_pos v0.0.0 (/checkout/src/libsyntax_pos)
[00:07:56]    Compiling rustc_errors v0.0.0 (/checkout/src/librustc_errors)
[00:08:06] error[E0277]: `syntax_pos::span_encoding::Span` cannot be shared between threads safely
[00:08:06]   --> src/libsyntax/tokenstream.rs:54:1
[00:08:06] 54 | / fn dummy()
[00:08:06] 55 | | where
[00:08:06] 56 | |     Span: Send + Sync,
[00:08:06] 56 | |     Span: Send + Sync,
[00:08:06] 57 | |     token::Token: Send + Sync,
[00:08:06] ...  |
[00:08:06] 60 | |     TokenStream: Send + Sync,
[00:08:06] 61 | | {}
[00:08:06]    | |__^ `syntax_pos::span_encoding::Span` cannot be shared between threads safely
[00:08:06]    = help: the trait `std::marker::Sync` is not implemented for `syntax_pos::span_encoding::Span`
[00:08:06]    = help: see issue #48214
[00:08:06]    = help: see issue #48214
[00:08:06]    = help: add #![feature(trivial_bounds)] to the crate attributes to enable
[00:08:06] 
[00:08:06] error[E0277]: `syntax_pos::span_encoding::Span` cannot be sent between threads safely
[00:08:06]   --> src/libsyntax/tokenstream.rs:54:1
[00:08:06] 54 | / fn dummy()
[00:08:06] 55 | | where
[00:08:06] 56 | |     Span: Send + Sync,
[00:08:06] 56 | |     Span: Send + Sync,
[00:08:06] 57 | |     token::Token: Send + Sync,
[00:08:06] ...  |
[00:08:06] 60 | |     TokenStream: Send + Sync,
[00:08:06] 61 | | {}
[00:08:06]    | |__^ `syntax_pos::span_encoding::Span` cannot be sent between threads safely
[00:08:06]    = help: the trait `std::marker::Send` is not implemented for `syntax_pos::span_encoding::Span`
[00:08:06]    = help: see issue #48214
[00:08:06]    = help: see issue #48214
[00:08:06]    = help: add #![feature(trivial_bounds)] to the crate attributes to enable
[00:08:06] 
[00:08:06] error[E0277]: `std::rc::Rc<parse::token::Nonterminal>` cannot be shared between threads safely
[00:08:06]   --> src/libsyntax/tokenstream.rs:54:1
[00:08:06] 54 | / fn dummy()
[00:08:06] 55 | | where
[00:08:06] 56 | |     Span: Send + Sync,
[00:08:06] 56 | |     Span: Send + Sync,
[00:08:06] 57 | |     token::Token: Send + Sync,
[00:08:06] ...  |
[00:08:06] 60 | |     TokenStream: Send + Sync,
[00:08:06] 61 | | {}
[00:08:06]    | |__^ `std::rc::Rc<parse::token::Nonterminal>` cannot be shared between threads safely
[00:08:06]    |
[00:08:06]    = help: within `parse::token::Token`, the trait `std::marker::Sync` is not implemented for `std::rc::Rc<parse::token::Nonterminal>`
[00:08:06]    = note: required because it appears within the type `parse::token::Token`
[00:08:06]    = help: see issue #48214
[00:08:06]    = help: add #![feature(trivial_bounds)] to the crate attributes to enable
[00:08:06] 
[00:08:06] error[E0277]: `std::rc::Rc<parse::token::Nonterminal>` cannot be sent between threads safely
[00:08:06]   --> src/libsyntax/tokenstream.rs:54:1
[00:08:06] 54 | / fn dummy()
[00:08:06] 55 | | where
[00:08:06] 56 | |     Span: Send + Sync,
[00:08:06] 56 | |     Span: Send + Sync,
[00:08:06] 57 | |     token::Token: Send + Sync,
[00:08:06] ...  |
[00:08:06] 60 | |     TokenStream: Send + Sync,
[00:08:06] 61 | | {}
[00:08:06]    | |__^ `std::rc::Rc<parse::token::Nonterminal>` cannot be sent between threads safely
[00:08:06]    |
[00:08:06]    = help: within `parse::token::Token`, the trait `std::marker::Send` is not implemented for `std::rc::Rc<parse::token::Nonterminal>`
[00:08:06]    = note: required because it appears within the type `parse::token::Token`
[00:08:06]    = help: see issue #48214
[00:08:06]    = help: add #![feature(trivial_bounds)] to the crate attributes to enable
[00:08:06] 
[00:08:06] error[E0277]: `std::rc::Rc<std::vec::Vec<(tokenstream::TokenTree, tokenstream::IsJoint)>>` cannot be shared between threads safely
[00:08:06]   --> src/libsyntax/tokenstream.rs:54:1
[00:08:06] 54 | / fn dummy()
[00:08:06] 55 | | where
[00:08:06] 56 | |     Span: Send + Sync,
[00:08:06] 56 | |     Span: Send + Sync,
[00:08:06] 57 | |     token::Token: Send + Sync,
[00:08:06] ...  |
[00:08:06] 60 | |     TokenStream: Send + Sync,
[00:08:06] 61 | | {}
[00:08:06]    | |__^ `std::rc::Rc<std::vec::Vec<(tokenstream::TokenTree, tokenstream::IsJoint)>>` cannot be shared between threads safely
[00:08:06]    |
[00:08:06]    = help: within `tokenstream::TokenStream`, the trait `std::marker::Sync` is not implemented for `std::rc::Rc<std::vec::Vec<(tokenstream::TokenTree, tokenstream::IsJoint)>>`
[00:08:06]    = note: required because it appears within the type `std::option::Option<std::rc::Rc<std::vec::Vec<(tokenstream::TokenTree, tokenstream::IsJoint)>>>`
[00:08:06]    = note: required because it appears within the type `tokenstream::TokenStream`
[00:08:06]    = help: see issue #48214
[00:08:06]    = help: add #![feature(trivial_bounds)] to the crate attributes to enable
[00:08:06] 
[00:08:06] error[E0277]: `std::rc::Rc<std::vec::Vec<(tokenstream::TokenTree, tokenstream::IsJoint)>>` cannot be sent between threads safely
[00:08:06]   --> src/libsyntax/tokenstream.rs:54:1
[00:08:06] 54 | / fn dummy()
[00:08:06] 55 | | where
[00:08:06] 56 | |     Span: Send + Sync,
[00:08:06] 56 | |     Span: Send + Sync,
[00:08:06] 57 | |     token::Token: Send + Sync,
[00:08:06] ...  |
[00:08:06] 60 | |     TokenStream: Send + Sync,
[00:08:06] 61 | | {}
[00:08:06]    | |__^ `std::rc::Rc<std::vec::Vec<(tokenstream::TokenTree, tokenstream::IsJoint)>>` cannot be sent between threads safely
[00:08:06]    |
[00:08:06]    = help: within `tokenstream::TokenStream`, the trait `std::marker::Send` is not implemented for `std::rc::Rc<std::vec::Vec<(tokenstream::TokenTree, tokenstream::IsJoint)>>`
[00:08:06]    = note: required because it appears within the type `std::option::Option<std::rc::Rc<std::vec::Vec<(tokenstream::TokenTree, tokenstream::IsJoint)>>>`
[00:08:06]    = note: required because it appears within the type `tokenstream::TokenStream`
[00:08:06]    = help: see issue #48214
[00:08:06]    = help: add #![feature(trivial_bounds)] to the crate attributes to enable
[00:08:06] error: aborting due to 6 previous errors
[00:08:06] 
[00:08:06] For more information about this error, try `rustc --explain E0277`.
[00:08:06] error: Could not compile `syntax`.
---
205996 ./obj/build/cache/2019-04-11
157460 ./obj/build/bootstrap/debug/incremental
156496 ./src/llvm-project/clang
142472 ./obj/build/bootstrap/debug/incremental/bootstrap-gm2kk8y15os9
142468 ./obj/build/bootstrap/debug/incremental/bootstrap-gm2kk8y15os9/s-fcda8kt4rb-umvqm1-34hadj2q5a3fu
108532 ./src/llvm-project/lldb
107952 ./obj/build/x86_64-unknown-linux-gnu/stage0-rustc
101812 ./.git
97584 ./src/llvm-project/clang/test

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@Zoxc Zoxc force-pushed the Zoxc:fix-syntax-sync branch from 220050f to 923369c May 19, 2019

@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented May 19, 2019

The job mingw-check of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:012bed40:start=1558294773169039585,finish=1558294773964235677,duration=795196092
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
$ export AWS_ACCESS_KEY_ID=AKIA46X5W6CZEJZ6XT55
---
[00:05:23]    Compiling synstructure v0.10.1
[00:05:37]    Compiling rustc_macros v0.1.0 (/checkout/src/librustc_macros)
[00:05:44]     Checking syntax_pos v0.0.0 (/checkout/src/libsyntax_pos)
[00:05:45]     Checking rustc_errors v0.0.0 (/checkout/src/librustc_errors)
[00:05:59] error: function is never used: `dummy`
[00:05:59]   --> src/libsyntax/tokenstream.rs:55:1
[00:05:59] 55 | / fn dummy()
[00:05:59] 56 | | where
[00:05:59] 57 | |     Span: Send + Sync,
[00:05:59] 57 | |     Span: Send + Sync,
[00:05:59] 58 | |     token::Token: Send + Sync,
[00:05:59] ...  |
[00:05:59] 61 | |     TokenStream: Send + Sync,
[00:05:59] 62 | | {}
[00:05:59]    |
[00:05:59]    = note: `-D dead-code` implied by `-D warnings`
[00:05:59] 
[00:06:00] error: aborting due to previous error
---
travis_time:end:0366d711:start=1558295146392795974,finish=1558295146398991039,duration=6195065
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:065171c8
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:1829528a
travis_time:start:1829528a
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:2dc5c9e0
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@michaelwoerister

This comment has been minimized.

Copy link
Contributor

commented May 21, 2019

Do we know the actual cause of the problem? Is the compiler running into some kind of livelock?

@michaelwoerister

This comment has been minimized.

Copy link
Contributor

commented May 21, 2019

This is what perf tells me after running ./x.py build for an hour or so:

Samples: 22M of event 'cycles:uppp', Event count (approx.): 22349476896140
Overhead  Command          Shared Object                                 Symbol
  21.14%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::traits::select::SelectionContext::evaluate_stack
   6.54%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::ty::query::plumbing::<impl rustc::ty::context::TyCtxt>::get_query
   4.53%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::traits::select::SelectionContext::evaluate_predicate_recursively
   3.03%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::ty::context::TyCtxt::_intern_substs
   2.51%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::infer::InferCtxt::start_snapshot
   2.48%  rustc            libc-2.28.so                                  [.] __memmove_avx_unaligned_erms
   2.31%  rustc            libc-2.28.so                                  [.] _int_malloc
   1.63%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::traits::select::SelectionContext::evaluate_predicates_recursively
   1.59%  rustc            libc-2.28.so                                  [.] _int_free
   1.34%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::infer::InferCtxt::in_snapshot
   1.25%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::ty::query::plumbing::<impl rustc::ty::context::TyCtxt>::get_query
   1.22%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::traits::select::SelectionContext::confirm_candidate
   1.10%  rustc            libc-2.28.so                                  [.] malloc
   1.00%  rustc            librustc-e9b5ca1e357beddc.so                  [.] <smallvec::SmallVec<A> as core::iter::traits::collect::FromIterator<<A as smallvec::Array>::Item>>::from_iter
   0.97%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::infer::InferCtxt::commit_from
   0.92%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::traits::select::SelectionContext::candidate_from_obligation
   0.88%  rustc            librustc-e9b5ca1e357beddc.so                  [.] <core::iter::adapters::flatten::FlatMap<I,U,F> as core::iter::traits::iterator::Iterator>::next
   0.86%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::ty::context::CtxtInterners::intern_ty
   0.82%  rustc            librustc-e9b5ca1e357beddc.so                  [.] <core::iter::adapters::flatten::FlatMap<I,U,F> as core::iter::traits::iterator::Iterator>::next
   0.81%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::ty::query::plumbing::<impl rustc::ty::context::TyCtxt>::get_query
   0.70%  rustc            libc-2.28.so                                  [.] realloc
   0.67%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::ty::fold::<impl rustc::ty::context::TyCtxt>::replace_bound_vars
   0.65%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::traits::project::normalize_with_depth
   0.62%  rustc            librustc-e9b5ca1e357beddc.so                  [.] <smallvec::SmallVec<A> as core::iter::traits::collect::FromIterator<<A as smallvec::Array>::Item>>::from_iter
   0.62%  rustc            librustc-e9b5ca1e357beddc.so                  [.] <rustc::traits::ObligationCause as core::clone::Clone>::clone
   0.60%  rustc            libLLVM-8-rust-1.35.0-beta.so                 [.] llvm::GlobalsAAResult::DeletionCallbackHandle::deleted
   0.58%  rustc            librustc-e9b5ca1e357beddc.so                  [.] <T as rustc::ty::context::InternIteratorElement<T,R>>::intern_with
   0.58%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::infer::InferCtxt::probe
   0.57%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::traits::select::SelectionContext::impl_or_trait_obligations
   0.49%  rustc            rustc                                         [.] free
   0.48%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::infer::InferCtxt::in_snapshot
   0.45%  rustc            librustc-e9b5ca1e357beddc.so                  [.] rustc::traits::select::SelectionContext::collect_predicates_for_types
   0.45%  rustc            librustc-e9b5ca1e357beddc.so                  [.] <rustc::traits::ObligationCause as core::clone::Clone>::clone
   0.40%  rustc            libLLVM-8-rust-1.35.0-beta.so                 [.] combineInstructionsOverFunction
   0.38%  rustc            libLLVM-8-rust-1.35.0-beta.so                 [.] llvm::ValueHandleBase::AddToUseList
   0.37%  rustc            rustc                                         [.] malloc
@pietroalbini

This comment has been minimized.

Copy link
Member

commented May 24, 2019

Beta nominating this. It's blocking the bootstrap update and the version number bump for nightly 1.37.0.

@Zoxc

This comment has been minimized.

Copy link
Contributor Author

commented May 24, 2019

We just need this on master to bump the bootstrap compiler.

@michaelwoerister

This comment has been minimized.

Copy link
Contributor

commented May 24, 2019

Let's land this for now. @Zoxc, r=me if you add an explanation why the explicit impls for TokenTree are needed (with a link to the GH issue).

@howaboutsynergy

This comment has been minimized.

Copy link

commented May 25, 2019

worked for me with commit 315ab95 (HEAD -> master, origin/master, origin/HEAD)

but without this patch I get infinite loop(?) for 4 hours #61162

Could the root cause of the infinite loop be fixed insteadalso?
Because this seems like a workaround. ie. it should maybe detect and fail without this patch.

@pietroalbini

This comment has been minimized.

Copy link
Member

commented May 25, 2019

This is blocking making nightly 1.37.0 so I'd prefer for the workaround to land first.

@Zoxc Zoxc force-pushed the Zoxc:fix-syntax-sync branch from 1c9bc50 to 3ed0561 May 27, 2019

@Zoxc

This comment has been minimized.

Copy link
Contributor Author

commented May 27, 2019

@bors r=michaelwoerister

@bors

This comment has been minimized.

Copy link
Contributor

commented May 27, 2019

📌 Commit 3ed0561 has been approved by michaelwoerister

@pietroalbini

This comment has been minimized.

Copy link
Member

commented May 27, 2019

@bors p=1

Blocking the nightly version bump.

@bors

This comment has been minimized.

Copy link
Contributor

commented May 27, 2019

⌛️ Testing commit 3ed0561 with merge 4dbc7f9...

bors added a commit that referenced this pull request May 27, 2019

Auto merge of #60967 - Zoxc:fix-syntax-sync, r=michaelwoerister
Short circuit Send and Sync impls for TokenTree

Workaround to make the parallel compiler build after #60444.

r? @nikomatsakis
@bors

This comment has been minimized.

Copy link
Contributor

commented May 27, 2019

☀️ Test successful - checks-travis, status-appveyor
Approved by: michaelwoerister
Pushing 4dbc7f9 to master...

@bors bors added the merged-by-bors label May 27, 2019

@bors bors merged commit 3ed0561 into rust-lang:master May 27, 2019

2 checks passed

Travis CI - Pull Request Build Passed
Details
homu Test successful
Details
@rust-highfive

This comment has been minimized.

Copy link
Collaborator

commented May 27, 2019

📣 Toolstate changed by #60967!

Tested on commit 4dbc7f9.
Direct link to PR: #60967

🎉 rls on windows: test-fail → test-pass (cc @Xanewok, @rust-lang/infra).

rust-highfive added a commit to rust-lang-nursery/rust-toolstate that referenced this pull request May 27, 2019

📣 Toolstate changed by rust-lang/rust#60967!
Tested on commit rust-lang/rust@4dbc7f9.
Direct link to PR: <rust-lang/rust#60967>

🎉 rls on windows: test-fail → test-pass (cc @Xanewok, @rust-lang/infra).
@pietroalbini

This comment has been minimized.

Copy link
Member

commented May 27, 2019

Removing the beta nomination.

@Zoxc Zoxc deleted the Zoxc:fix-syntax-sync branch May 27, 2019

@Keruspe

This comment has been minimized.

Copy link
Contributor

commented May 28, 2019

@pietroalbini why removing the beta nomination? I guess this could be the cause of #61262 ?

@pnkfelix

This comment has been minimized.

Copy link
Member

commented Jun 4, 2019

I think @pietroalbini removed the beta nomination because we typically reserve beta backports for things that will help resolve "customer" issues, and entities bootstrapping the compiler (be theyrustc developers or package distribution systems) are, for better or for worse, not always considered first-class customers?

(I am not trying to disparage anyone in my comment above; I made the above statement as someone who both is a rustc developer bootstrapping the compiler all the time, and also as someone who triages bugs on a weekly basis and struggles with the question of which bugs are worth the risk of backporting to beta.)

Anyway, what would be the expected utility of a beta-backport here? As far as I can tell by reading the discussion on #61262, the problem only arises when one uses parallel-compiler=true for a bootstrap, right? Are you bootstrapping using a checkout of the source code of the beta-branch itself?

@Keruspe

This comment has been minimized.

Copy link
Contributor

commented Jun 4, 2019

@pnkfelix I am the rust packager for Exherbo and the package builds the beta branch directly from a checkout.
I closed the bug though as the issue can indeed be worked around

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.