Skip to content

/rust

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store a Symbol in InternedString #46972

Closed
wants to merge 2 commits into from

Conversation

Reviewers
No reviews
Assignees

@jseyfried jseyfried

Labels
S-inactive-closed T-compiler
Projects
None yet
Milestone
No milestone
@Zoxc
Copy link
Contributor

Zoxc commented Dec 23, 2017

No description provided.

@rust-highfive rust-highfive assigned pnkfelix Dec 23, 2017

@rust-highfive

This comment has been minimized.

Sign in to view
Copy link
Collaborator

rust-highfive commented Dec 23, 2017

r? @pnkfelix

(rust_highfive has picked a reviewer for you, use r? to override)

@Zoxc Zoxc force-pushed the Zoxc:internedstring-no-deref branch from c758e13 to 100c61b Dec 24, 2017

@eddyb

This comment has been minimized.

Sign in to view
Copy link
Member

eddyb commented Dec 24, 2017 •
edited

r? @jseyfried (I don't think we want these changes but we probably need a wider discussion)

EDIT: InternedString, before this PR, might be unsound, wrt placing it in a thread_local.
However, that wouldn't be the case if the interner is owned outside of the threads using it.

@rust-highfive rust-highfive assigned jseyfried and unassigned pnkfelix Dec 24, 2017

@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Dec 25, 2017

☔️ The latest upstream changes (presumably #46899) made this pull request unmergeable. Please resolve the merge conflicts.

@Zoxc Zoxc force-pushed the Zoxc:internedstring-no-deref branch from 100c61b to 4cda813 Dec 25, 2017

@Zoxc Zoxc changed the title WIP: Store a Symbol in InternedString Store a Symbol in InternedString Dec 25, 2017

@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Dec 27, 2017

☔️ The latest upstream changes (presumably #46803) made this pull request unmergeable. Please resolve the merge conflicts.

@kennytm kennytm added the S-waiting-on-review label Dec 27, 2017

@Zoxc Zoxc force-pushed the Zoxc:internedstring-no-deref branch from 4cda813 to c3ab779 Dec 28, 2017

@Zoxc

This comment has been minimized.

Sign in to view
Copy link
Contributor Author

Zoxc commented Dec 28, 2017

@bors try

bors added a commit that referenced this pull request Dec 28, 2017

@bors
Auto merge of #46972 - Zoxc:internedstring-no-deref, r=<try> 
Store a Symbol in InternedString
Loading status checks…
8c52267
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Dec 28, 2017

⌛️ Trying commit c3ab779 with merge 8c52267...
@bors

This comment has been minimized.

Sign in to view
Copy link
Contributor

bors commented Dec 28, 2017

☀️ Test successful - status-travis
State: approved= try=True
@Zoxc

This comment has been minimized.

Sign in to view
Copy link
Contributor Author

Zoxc commented Dec 28, 2017

@Mark-Simulacrum I'd like a perf run on this
@Mark-Simulacrum

This comment has been minimized.

Sign in to view
Copy link
Member

Mark-Simulacrum commented Dec 29, 2017

Perf run queued.
@alexcrichton

This comment has been minimized.

Sign in to view
Copy link
Member

alexcrichton commented Jan 4, 2018

ping @Mark-Simulacrum, do we have perf results now perchance?
@pnkfelix

This comment has been minimized.

Sign in to view
Copy link
Member

pnkfelix commented Jan 4, 2018

(Also, Note that @jseyfried has not yet weighed in on whether they want these changes.)
@Mark-Simulacrum
@carols10cents

This comment has been minimized.

Sign in to view
Copy link
Member

carols10cents commented Jan 9, 2018 •
edited

review ping @jseyfried ! pinging you on IRC too! or not because i don't see you :)

@kennytm kennytm added the T-compiler label Jan 17, 2018

@kennytm

This comment has been minimized.

Sign in to view
Copy link
Member

kennytm commented Jan 17, 2018

Review ping for you @jseyfried! Also cc @rust-lang/compiler.
@nikomatsakis

This comment has been minimized.

Sign in to view
Copy link
Contributor

nikomatsakis commented Jan 17, 2018

Can somebody summarize (a) what is being done here so I don't have to read the diffs and (b) why I should care?
@eddyb

This comment has been minimized.

Sign in to view
Copy link
Member

eddyb commented Jan 17, 2018 •
edited

@nikomatsakis This is reverting an API change @jseyfried (I believe) did.
Right now we deref InternedString to str (sound because InternedString is not Send/Sync).
@Zoxc moves us backwards to an with_str style API with this PR (which makes InternedString largely redundant with Symbol - IMO InternedString should be e.g. SymbolContents anyway).
@petrochenkov

This comment has been minimized.

Sign in to view
Copy link
Contributor

petrochenkov commented Jan 17, 2018

What's the point of InternedString as a separate type if it's equivalent to Symbol now?

So far I didn't look carefully at @Zoxc's multithreading-related PRs and mostly seen them as something quite unfortunate, but maybe necessary after all.
Could someone write a design summary about how Symbols are supposed to be organized in multithreaded rustc?
Is there going to be a global interner? Or many thread-local interners with string duplication? Or both?
@Zoxc

This comment has been minimized.

Sign in to view
Copy link
Contributor Author

Zoxc commented Jan 17, 2018

What's the point of InternedString as a separate type if it's equivalent to Symbol now?

It has Ord and Hash implementations based on strings instead of an unstable integer like Symbol.

Is there going to be a global interner? Or many thread-local interners with string duplication? Or both?

My branch has a interner per compilation session which is behind a lock. There is no parallelism during parsing there though, so all symbols are identical to what happens on master.
@michaelwoerister

This comment has been minimized.

Sign in to view
Copy link
Contributor

michaelwoerister commented Jan 18, 2018

This seems to be slowing things down quite a bit (as per the perf run above). Is this a soundness fix? Is there an alternative implementation that allows us to keep using direct string references?
@eddyb

This comment has been minimized.

Sign in to view
Copy link
Member

eddyb commented Jan 18, 2018

I believe we can just require that the interner outlives the threads that use it.
@Zoxc

This comment has been minimized.

Sign in to view
Copy link
Contributor Author

Zoxc commented Jan 18, 2018

@michaelwoerister I suspect the slowdown is due to the usages of to_string() here. Most of these are because we want to match the interned string on constant strings and there's some return inside so we don't want a closure. We should instead pre-intern these constant strings using a macro. This avoids the string allocation and the string comparisons.

@eddyb I think that spawning a thread for the interner should work. Not terribly elegant though.
@eddyb

This comment has been minimized.

Sign in to view
Copy link
Member

eddyb commented Jan 18, 2018

@Zoxc I mean you already have the threads, you just have to own the interner outside of them.

@carols10cents carols10cents added S-waiting-on-author and removed S-waiting-on-review labels Jan 22, 2018

@nikomatsakis

This comment has been minimized.

Sign in to view
Copy link
Contributor

nikomatsakis commented Jan 25, 2018

We should instead pre-intern these constant strings using a macro. This avoids the string allocation and the string comparisons.

It seems pretty important to resolve this, no?
@shepmaster

This comment has been minimized.

Sign in to view
Copy link
Member

shepmaster commented Feb 3, 2018

Thanks for the PR, @Zoxc ! Since we haven't heard from you in a few weeks, I'm going to close this for now. Please feel free to reopen once you address the merge conflicts and the last round of feedback!

@shepmaster shepmaster closed this Feb 3, 2018

@shepmaster shepmaster added S-inactive-closed and removed S-waiting-on-author labels Feb 3, 2018

@petrochenkov petrochenkov referenced this pull request Mar 22, 2018

Closed

Use InternedString instead of Symbol for type parameter types #49266

This was referenced Apr 9, 2018

Merged
Merged

bors added a commit that referenced this pull request Apr 20, 2018

@bors
Auto merge of #49894 - Zoxc:sync-internedstring, r=<try> 
Rename InternedString to LocalInternedString and introduce a new thread-safe InternedString

This is an allocation-free alternative to #46972.

cc @rust-lang/compiler

r? @michaelwoerister
Loading status checks…
7ef4d38

kennytm added a commit to kennytm/rust that referenced this pull request Apr 27, 2018

@kennytm
Rollup merge of rust-lang#49894 - Zoxc:sync-internedstring, r=michael… 
…woerister

Rename InternedString to LocalInternedString and introduce a new thread-safe InternedString

This is an allocation-free alternative to rust-lang#46972.
Verified
This commit was signed with a verified signature.
@kennytm kennytm
GPG key ID: FEF6C8051D0E013C Learn about signing commits
44b5359

alexcrichton added a commit to alexcrichton/rust that referenced this pull request May 10, 2018

@alexcrichton
Rollup merge of rust-lang#49823 - Zoxc:term-str, r=alexcrichton 
Remove usages of Term::as_str and mark it for removal

Returning references to rustc internal data structures is a bad idea since their lifetimes are unrelated to the lifetimes of proc_macro values.

See rust-lang#46972 and the `Taming thread-local storage` section of https://internals.rust-lang.org/t/parallelizing-rustc-using-rayon/6606

r? @alexcrichton
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
54018b1

bors added a commit that referenced this pull request May 10, 2018

@bors
Auto merge of #49823 - Zoxc:term-str, r=alexcrichton 
Remove usages of Term::as_str and mark it for removal

Returning references to rustc internal data structures is a bad idea since their lifetimes are unrelated to the lifetimes of proc_macro values.

See #46972 and the `Taming thread-local storage` section of https://internals.rust-lang.org/t/parallelizing-rustc-using-rayon/6606

r? @alexcrichton
Loading status checks…
9e3caa2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.