Work towards thread safety in rustc #46779

Zoxc · 2017-12-17T01:38:19Z

This PR is split out from #45912. It contains changes which do not require the sync module.

rust-highfive · 2017-12-17T01:38:24Z

(rust_highfive has picked a reviewer for you, use r? to override)

bjorn3 · 2017-12-17T11:09:40Z

src/libserialize/collection_impls.rs

+            Ok(vec.into())
+        })
+    }
+}


Missing trailing newline

bjorn3 · 2017-12-17T11:13:54Z

src/libsyntax_pos/lib.rs

@@ -163,10 +163,6 @@ impl SpanData {
    }
 }

-// The interner in thread-local, so `Span` shouldn't move between threads.
-impl !Send for Span {}


Could you please make the interner thread-safe?

bjorn3 · 2017-12-17T11:13:58Z

src/libsyntax_pos/symbol.rs

@@ -83,10 +83,6 @@ impl Decodable for Ident {
 #[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
 pub struct Symbol(u32);

-// The interner in thread-local, so `Symbol` shouldn't move between threads.
-impl !Send for Symbol { }


Zoxc · 2017-12-17T13:17:20Z

@bjorn3 Thread-safety for the interners happens in a later commit.

bjorn3 · 2017-12-17T13:27:00Z

@Zoxc that commit doesnt seem to be included in this pr.

Zoxc · 2017-12-17T13:29:01Z

It is in #45912.

michaelwoerister · 2017-12-18T15:33:10Z

Each PR must leave the compiler in safe/valid state. Implementing Send/Sync for Symbol without making the interners threadsafe is not an option.

Zoxc · 2017-12-18T16:03:59Z

@michaelwoerister The !Send impls are not there for thread safety, but as a way to help ensure that Symbol types are only used with the interner they are created from. This trick no longer works when multiple threads share an interner. It also doesn't work if there are multiple interners per thread (which some of my upcoming patches allow).

michaelwoerister · 2017-12-18T16:41:35Z

My concern is just that this PR, as it is, would leave the compiler in a state where I can send Symbols between threads and thus get wrong results from the auto-generated Eq implementation. The !Send implementation should only be removed if the same PR makes sure that that cannot lead to logic errors.

arielb1 · 2017-12-18T17:59:43Z

src/librustc_errors/lib.rs

@@ -42,6 +42,8 @@ use std::cell::{RefCell, Cell};
 use std::mem;
 use std::rc::Rc;
 use std::{error, fmt};
+use std::sync::atomic::AtomicUsize;


err_count probably needs to be done in a smarter way than here to assure thread-safety, so could you add a FIXME? In any case this is better than the way before this.

In which case is this not thread safe?

arielb1 · 2017-12-18T18:03:09Z

src/librustc/ich/caching_codemap_view.rs

-            let files = self.codemap.files();
-
-            if files.len() > 0 {
+            if self.codemap.files().len() > 0 {


Don't we need a better locking discipline for the codemap? It feels like "too many locks, not enough discipline". Maybe just add a fixme to it?

Given that I just convert all RefCells into locks, there's probably plenty of places where there are too many locks.

arielb1 · 2017-12-18T18:07:29Z

src/libsyntax_pos/lib.rs

@@ -163,10 +163,6 @@ impl SpanData {
    }


How is this sound?

arielb1 · 2017-12-18T18:07:42Z

src/libsyntax_pos/symbol.rs

@@ -384,8 +384,6 @@ impl<'a> ::std::cmp::PartialEq<InternedString> for &'a String {
    }


How is this sound?

Oh, InternedString storing a &'static str isn't sound at all. I'll have to change it to using a u32 like Symbol.

Well that made it unable to implement Deref<Target=str>, so I went with the approach suggested in a FIXME and just leak interned strings.

Zoxc · 2017-12-21T18:23:08Z

@bors r=arielb1

bors · 2017-12-21T18:23:09Z

📌 Commit 84ce4f1 has been approved by arielb1

bors · 2017-12-22T12:34:51Z

⌛ Testing commit 84ce4f1 with merge 2c037d5...

Work towards thread safety in rustc This PR is split out from #45912. It contains changes which do not require the `sync` module.

bors · 2017-12-22T15:15:14Z

☀️ Test successful - status-appveyor, status-travis
Approved by: arielb1
Pushing 2c037d5 to master...

eddyb · 2017-12-22T15:35:00Z

src/libsyntax/attr.rs

-        slot.set(r + 1);
-        r
-    });
+    let id = NEXT_ATTR_ID.fetch_add(1, Ordering::SeqCst);


@rust-lang/compiler Not sure how I feel about this kind of stuff - it will behave non-deterministically in any process running multiple rustc threads in parallel (atm rustc "owns" a thread).
IMO all thread-locals should change to scoped thread locals initialized to point to some memory owned by the thread that started the compilation, if we want multiple threads per rustc "instance".

Good point. I'll make this either a scoped thread local or put it in ParseSess if that happens to be easy.

rust-highfive assigned estebank Dec 17, 2017

Zoxc assigned arielb1 and unassigned estebank Dec 17, 2017

Zoxc force-pushed the par-merge-without-sync branch from c67d30e to 6b295aa Compare December 17, 2017 01:58

kennytm added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Dec 17, 2017

bjorn3 requested changes Dec 17, 2017

View reviewed changes

arielb1 reviewed Dec 18, 2017

View reviewed changes

src/libsyntax_pos/lib.rs Outdated

@@ -163,10 +163,6 @@ impl SpanData {

}

Copy link

Contributor

arielb1 Dec 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is this sound?

arielb1 reviewed Dec 18, 2017

View reviewed changes

Zoxc force-pushed the par-merge-without-sync branch from 6b295aa to dbee5b9 Compare December 19, 2017 01:00

Zoxc added 8 commits December 21, 2017 19:21

Combine GlobalArenas and DroplessArena into AllArenas

d81cd38

Make mk_attr_id thread safe

70fd306

Make err_count thread safe

94b7124

Refactor code so the call to codemap.files() does not deadlock

f7082dc

Add a -Z query-threads compiler option

aa6065b

Make IndexVec implement Send and Sync

30a39ac

Remove useless Rc

30733b3

Add Encodable and Decodable impls for Arc<[T]>

84ce4f1

Zoxc force-pushed the par-merge-without-sync branch from dbee5b9 to 84ce4f1 Compare December 21, 2017 18:22

bors added a commit that referenced this pull request Dec 22, 2017

Auto merge of #46779 - Zoxc:par-merge-without-sync, r=arielb1

2c037d5

Work towards thread safety in rustc This PR is split out from #45912. It contains changes which do not require the `sync` module.

bors merged commit 84ce4f1 into rust-lang:master Dec 22, 2017

bors mentioned this pull request Dec 22, 2017

Allow runtime switching between trans backends #45684

Merged

eddyb reviewed Dec 22, 2017

View reviewed changes

aturon mentioned this pull request Aug 16, 2019

Audit sources of shared state across rustc #63643

Open

37 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Work towards thread safety in rustc #46779

Work towards thread safety in rustc #46779

Zoxc commented Dec 17, 2017

rust-highfive commented Dec 17, 2017

bjorn3 Dec 17, 2017

bjorn3 Dec 17, 2017

bjorn3 Dec 17, 2017

Zoxc commented Dec 17, 2017

bjorn3 commented Dec 17, 2017

Zoxc commented Dec 17, 2017

michaelwoerister commented Dec 18, 2017

Zoxc commented Dec 18, 2017

michaelwoerister commented Dec 18, 2017

arielb1 Dec 18, 2017 •

edited

Loading

Zoxc Dec 19, 2017

arielb1 Dec 18, 2017

Zoxc Dec 19, 2017

arielb1 Dec 21, 2017

arielb1 Dec 18, 2017

arielb1 Dec 18, 2017

Zoxc Dec 18, 2017

Zoxc Dec 19, 2017

Zoxc commented Dec 21, 2017

bors commented Dec 21, 2017

bors commented Dec 22, 2017

bors commented Dec 22, 2017

eddyb Dec 22, 2017

Zoxc Dec 22, 2017

		@@ -384,8 +384,6 @@ impl<'a> ::std::cmp::PartialEq<InternedString> for &'a String {
		}

Work towards thread safety in rustc #46779

Work towards thread safety in rustc #46779

Conversation

Zoxc commented Dec 17, 2017

rust-highfive commented Dec 17, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Zoxc commented Dec 17, 2017

bjorn3 commented Dec 17, 2017

Zoxc commented Dec 17, 2017

michaelwoerister commented Dec 18, 2017

Zoxc commented Dec 18, 2017

michaelwoerister commented Dec 18, 2017

arielb1 Dec 18, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Zoxc commented Dec 21, 2017

bors commented Dec 21, 2017

bors commented Dec 22, 2017

bors commented Dec 22, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arielb1 Dec 18, 2017 •

edited

Loading