Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid unnecessary `TokenTree` to `TokenStream` conversions #65455

Conversation

@nnethercote
Copy link
Contributor

nnethercote commented Oct 15, 2019

A TokenStream contains any number of TokenTrees. Therefore, a single TokenTree can be promoted to a TokenStream. But doing so costs two allocations: one for the single-element Vec, and one for the Lrc. (An IsJoint value also must be added; the default is NonJoint.)

The current code converts TokenTrees to TokenStreams unnecessarily in a few places. This PR removes some of these unnecessary conversions, both simplifying the code and speeding it up.

r? @petrochenkov

@nnethercote

This comment has been minimized.

Copy link
Contributor Author

nnethercote commented Oct 15, 2019

@bors try @rust-timer queue

@rust-timer

This comment has been minimized.

Copy link

rust-timer commented Oct 15, 2019

Awaiting bors try build completion

@bors

This comment has been minimized.

Copy link
Contributor

bors commented Oct 15, 2019

⌛️ Trying commit 08afe1f with merge 881f05b...

bors added a commit that referenced this pull request Oct 15, 2019
…enStream-conversions, r=<try>

Avoid unnecessary `TokenTree` to `TokenStream` conversions

A `TokenStream` contains any number of `TokenTrees`. Therefore, a single `TokenTree` can be promoted to a `TokenStream`. But doing so costs two allocations: one for the single-element `Vec`, and one for the `Lrc`. (An `IsJoint` value also must be added; the default is `NonJoint`.)

The current code converts `TokenTree`s to `TokenStream`s unnecessarily in a few places. This PR removes some of these unnecessary conversions, both simplifying the code and speeding it up.

r? @petrochenkov
@bors

This comment has been minimized.

Copy link
Contributor

bors commented Oct 16, 2019

☀️ Try build successful - checks-azure
Build commit: 881f05b (881f05b8576331a868132c5d3201bb7e54a2fff3)

@rust-timer

This comment has been minimized.

Copy link

rust-timer commented Oct 16, 2019

Queued 881f05b with parent 237d54f, future comparison URL.

src/libsyntax/attr/mod.rs Outdated Show resolved Hide resolved
@petrochenkov

This comment has been minimized.

Copy link
Contributor

petrochenkov commented Oct 16, 2019

r=me after the perf is ready, with or without addressing #65455 (comment)

@rust-timer

This comment has been minimized.

Copy link

rust-timer commented Oct 16, 2019

Finished benchmarking try commit 881f05b, comparison URL.

@mati865

This comment has been minimized.

Copy link
Contributor

mati865 commented Oct 16, 2019

In case perf.rlo dies again the results are good.

Screenshot

Screenshot_2019-10-16 rustc performance data

@nnethercote nnethercote force-pushed the nnethercote:avoid-unnecessary-TokenTree-to-TokenStream-conversions branch from 08afe1f to 0fb82f3 Oct 16, 2019
@nnethercote

This comment has been minimized.

Copy link
Contributor Author

nnethercote commented Oct 16, 2019

I added MetaItemKind::token() back in, because it's useful. But MetaItem::tokens() and NestedMetaItem::tokens() aren't necessary, so I didn't add them back in.

@bors r=petrochenkov

@bors

This comment has been minimized.

Copy link
Contributor

bors commented Oct 16, 2019

📌 Commit 0fb82f3 has been approved by petrochenkov

@bors

This comment has been minimized.

Copy link
Contributor

bors commented Oct 17, 2019

☔️ The latest upstream changes (presumably #65495) made this pull request unmergeable. Please resolve the merge conflicts.

bors added a commit that referenced this pull request Oct 17, 2019
Rollup of 7 pull requests

Successful merges:

 - #64890 ([const-prop] Handle remaining MIR Rvalue cases)
 - #65094 (Prefer statx on linux if available)
 - #65201 (Disable Go and OCaml bindings when building LLVM)
 - #65334 (Add long error explanation for E0575)
 - #65417 (Add more coherence tests)
 - #65496 (properly document panics in div_euclid and rem_euclid)
 - #65508 (add option to ping llvm ice-breakers to triagebot)

Failed merges:

 - #65390 (Add long error explanation for E0576)
 - #65434 (Add long error explanation for E0577)
 - #65455 (Avoid unnecessary `TokenTree` to `TokenStream` conversions)
 - #65471 (Add long error explanation for E0578)
 - #65501 (Remove `src/llvm-emscripten` submodule)

r? @ghost
nnethercote added 4 commits Oct 13, 2019
The current code has this impl:
```
impl<T: Into<TokenStream>> iter::FromIterator<T> for TokenStream
```
If given an `IntoIterator<Item = TokenTree>`, it will convert each individual
`TokenTree` to a `TokenStream` (at the cost of two allocations: a `Vec`
and an `Lrc`). It will then merge those `TokenStream`s into a single
`TokenStream`. This is inefficient.

This commit changes the impl to this less general one:
```
impl iter::FromIterator<TokenTree> for TokenStream
```
It collects the `TokenTree`s into a single `Vec` first and then converts that
to a `TokenStream` by wrapping it in a single `Lrc`. The previous generality
was unnecessary; no other code needs changing.

This change speeds up several benchmarks by up to 4%.
Because most of the call sites have an easier time working with a
`TokenTree` instead of a `TokenStream`.
Likewise for `NestedMetaItem::tokens()`. Also, add
`MetaItemKind::token_trees_and_joints()`, which `MetaItemKind::tokens()`
now calls.

This avoids some unnecessary `TokenTree` to `TokenStream` conversions,
and removes the need for the clumsy
`TokenStream::append_to_tree_and_joint_vec()`.
@nnethercote nnethercote force-pushed the nnethercote:avoid-unnecessary-TokenTree-to-TokenStream-conversions branch from 0fb82f3 to e4ec4a6 Oct 18, 2019
@nnethercote

This comment has been minimized.

Copy link
Contributor Author

nnethercote commented Oct 18, 2019

I rebased.

@bors r=petrochenkov

@bors

This comment has been minimized.

Copy link
Contributor

bors commented Oct 18, 2019

📌 Commit e4ec4a6 has been approved by petrochenkov

tmandry added a commit to tmandry/rust that referenced this pull request Oct 18, 2019
…Tree-to-TokenStream-conversions, r=petrochenkov

Avoid unnecessary `TokenTree` to `TokenStream` conversions

A `TokenStream` contains any number of `TokenTrees`. Therefore, a single `TokenTree` can be promoted to a `TokenStream`. But doing so costs two allocations: one for the single-element `Vec`, and one for the `Lrc`. (An `IsJoint` value also must be added; the default is `NonJoint`.)

The current code converts `TokenTree`s to `TokenStream`s unnecessarily in a few places. This PR removes some of these unnecessary conversions, both simplifying the code and speeding it up.

r? @petrochenkov
bors added a commit that referenced this pull request Oct 18, 2019
Rollup of 13 pull requests

Successful merges:

 - #64925 (Document JSON message output.)
 - #65201 (Disable Go and OCaml bindings when building LLVM)
 - #65334 (Add long error explanation for E0575)
 - #65417 (Add more coherence tests)
 - #65455 (Avoid unnecessary `TokenTree` to `TokenStream` conversions)
 - #65472 (Use a sharded dep node to dep node index map)
 - #65480 (Speed up `LexicalResolve::expansion()`)
 - #65493 (Add long error explanation for E0584)
 - #65496 (properly document panics in div_euclid and rem_euclid)
 - #65498 (Plugins deprecation: don’t suggest simply removing the attribute)
 - #65508 (add option to ping llvm ice-breakers to triagebot)
 - #65513 (reorder fmt docs for more clarity)
 - #65531 (Update backtrace to 0.3.40)

Failed merges:

 - #65390 (Add long error explanation for E0576)
 - #65434 (Add long error explanation for E0577)
 - #65471 (Add long error explanation for E0578)
 - #65501 (Remove `src/llvm-emscripten` submodule)

r? @ghost
tmandry added a commit to tmandry/rust that referenced this pull request Oct 18, 2019
…Tree-to-TokenStream-conversions, r=petrochenkov

Avoid unnecessary `TokenTree` to `TokenStream` conversions

A `TokenStream` contains any number of `TokenTrees`. Therefore, a single `TokenTree` can be promoted to a `TokenStream`. But doing so costs two allocations: one for the single-element `Vec`, and one for the `Lrc`. (An `IsJoint` value also must be added; the default is `NonJoint`.)

The current code converts `TokenTree`s to `TokenStream`s unnecessarily in a few places. This PR removes some of these unnecessary conversions, both simplifying the code and speeding it up.

r? @petrochenkov
bors added a commit that referenced this pull request Oct 18, 2019
Rollup of 12 pull requests

Successful merges:

 - #64925 (Document JSON message output.)
 - #65201 (Disable Go and OCaml bindings when building LLVM)
 - #65334 (Add long error explanation for E0575)
 - #65417 (Add more coherence tests)
 - #65455 (Avoid unnecessary `TokenTree` to `TokenStream` conversions)
 - #65472 (Use a sharded dep node to dep node index map)
 - #65480 (Speed up `LexicalResolve::expansion()`)
 - #65493 (Add long error explanation for E0584)
 - #65496 (properly document panics in div_euclid and rem_euclid)
 - #65498 (Plugins deprecation: don’t suggest simply removing the attribute)
 - #65508 (add option to ping llvm ice-breakers to triagebot)
 - #65513 (reorder fmt docs for more clarity)

Failed merges:

 - #65390 (Add long error explanation for E0576)
 - #65434 (Add long error explanation for E0577)
 - #65471 (Add long error explanation for E0578)
 - #65501 (Remove `src/llvm-emscripten` submodule)

r? @ghost
tmandry added a commit to tmandry/rust that referenced this pull request Oct 18, 2019
…Tree-to-TokenStream-conversions, r=petrochenkov

Avoid unnecessary `TokenTree` to `TokenStream` conversions

A `TokenStream` contains any number of `TokenTrees`. Therefore, a single `TokenTree` can be promoted to a `TokenStream`. But doing so costs two allocations: one for the single-element `Vec`, and one for the `Lrc`. (An `IsJoint` value also must be added; the default is `NonJoint`.)

The current code converts `TokenTree`s to `TokenStream`s unnecessarily in a few places. This PR removes some of these unnecessary conversions, both simplifying the code and speeding it up.

r? @petrochenkov
bors added a commit that referenced this pull request Oct 18, 2019
Rollup of 17 pull requests

Successful merges:

 - #65016 (Always inline `mem::{size_of,align_of}` in debug builds)
 - #65197 (Prepare `MutVisitor`s to handle interned projections)
 - #65201 (Disable Go and OCaml bindings when building LLVM)
 - #65364 (Collect occurrences of empty blocks for mismatched braces diagnostic)
 - #65417 (Add more coherence tests)
 - #65434 (Add long error explanation for E0577)
 - #65455 (Avoid unnecessary `TokenTree` to `TokenStream` conversions)
 - #65472 (Use a sharded dep node to dep node index map)
 - #65480 (Speed up `LexicalResolve::expansion()`)
 - #65496 (properly document panics in div_euclid and rem_euclid)
 - #65508 (add option to ping llvm ice-breakers to triagebot)
 - #65511 (save-analysis: Nest tables when processing impl block definitions)
 - #65513 (reorder fmt docs for more clarity)
 - #65532 (doc: make BitSet intro more short)
 - #65540 (show up some extra info when t!() fails)
 - #65549 (Fix left/right shift typo in wrapping rotate docs)
 - #65552 (Clarify diagnostics when using `~` as a unary op)

Failed merges:

 - #65471 (Add long error explanation for E0578)

r? @ghost
tmandry added a commit to tmandry/rust that referenced this pull request Oct 18, 2019
…Tree-to-TokenStream-conversions, r=petrochenkov

Avoid unnecessary `TokenTree` to `TokenStream` conversions

A `TokenStream` contains any number of `TokenTrees`. Therefore, a single `TokenTree` can be promoted to a `TokenStream`. But doing so costs two allocations: one for the single-element `Vec`, and one for the `Lrc`. (An `IsJoint` value also must be added; the default is `NonJoint`.)

The current code converts `TokenTree`s to `TokenStream`s unnecessarily in a few places. This PR removes some of these unnecessary conversions, both simplifying the code and speeding it up.

r? @petrochenkov
bors added a commit that referenced this pull request Oct 18, 2019
Rollup of 19 pull requests

Successful merges:

 - #65016 (Always inline `mem::{size_of,align_of}` in debug builds)
 - #65197 (Prepare `MutVisitor`s to handle interned projections)
 - #65201 (Disable Go and OCaml bindings when building LLVM)
 - #65334 (Add long error explanation for E0575)
 - #65364 (Collect occurrences of empty blocks for mismatched braces diagnostic)
 - #65455 (Avoid unnecessary `TokenTree` to `TokenStream` conversions)
 - #65472 (Use a sharded dep node to dep node index map)
 - #65480 (Speed up `LexicalResolve::expansion()`)
 - #65493 (Add long error explanation for E0584)
 - #65496 (properly document panics in div_euclid and rem_euclid)
 - #65498 (Plugins deprecation: don’t suggest simply removing the attribute)
 - #65508 (add option to ping llvm ice-breakers to triagebot)
 - #65511 (save-analysis: Nest tables when processing impl block definitions)
 - #65513 (reorder fmt docs for more clarity)
 - #65532 (doc: make BitSet intro more short)
 - #65535 (rustc: arena-allocate the slice in `ty::GenericsPredicate`, not the whole struct.)
 - #65540 (show up some extra info when t!() fails)
 - #65549 (Fix left/right shift typo in wrapping rotate docs)
 - #65552 (Clarify diagnostics when using `~` as a unary op)

Failed merges:

 - #65390 (Add long error explanation for E0576)
 - #65434 (Add long error explanation for E0577)
 - #65471 (Add long error explanation for E0578)

r? @ghost
bors added a commit that referenced this pull request Oct 19, 2019
Rollup of 19 pull requests

Successful merges:

 - #65016 (Always inline `mem::{size_of,align_of}` in debug builds)
 - #65197 (Prepare `MutVisitor`s to handle interned projections)
 - #65201 (Disable Go and OCaml bindings when building LLVM)
 - #65334 (Add long error explanation for E0575)
 - #65364 (Collect occurrences of empty blocks for mismatched braces diagnostic)
 - #65455 (Avoid unnecessary `TokenTree` to `TokenStream` conversions)
 - #65472 (Use a sharded dep node to dep node index map)
 - #65480 (Speed up `LexicalResolve::expansion()`)
 - #65493 (Add long error explanation for E0584)
 - #65496 (properly document panics in div_euclid and rem_euclid)
 - #65498 (Plugins deprecation: don’t suggest simply removing the attribute)
 - #65508 (add option to ping llvm ice-breakers to triagebot)
 - #65511 (save-analysis: Nest tables when processing impl block definitions)
 - #65513 (reorder fmt docs for more clarity)
 - #65532 (doc: make BitSet intro more short)
 - #65535 (rustc: arena-allocate the slice in `ty::GenericsPredicate`, not the whole struct.)
 - #65540 (show up some extra info when t!() fails)
 - #65549 (Fix left/right shift typo in wrapping rotate docs)
 - #65552 (Clarify diagnostics when using `~` as a unary op)

Failed merges:

 - #65390 (Add long error explanation for E0576)
 - #65434 (Add long error explanation for E0577)
 - #65471 (Add long error explanation for E0578)

r? @ghost
@bors bors merged commit e4ec4a6 into rust-lang:master Oct 19, 2019
@nnethercote nnethercote deleted the nnethercote:avoid-unnecessary-TokenTree-to-TokenStream-conversions branch Oct 20, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.