Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Avoid unnecessary TokenTree to TokenStream conversions #65455

Merged
merged 4 commits into from
Oct 19, 2019
Merged

Avoid unnecessary TokenTree to TokenStream conversions #65455

merged 4 commits into from
Oct 19, 2019

Commits on Oct 18, 2019

  1. Remove two no-op into() calls.

    @nnethercote
    nnethercote committed Oct 18, 2019
    Configuration menu
    Copy the full SHA
    d0eaf60 View commit details
    Browse the repository at this point in the history

  2. Make TokenStream::from_iter less general and more efficient. 

    The current code has this impl:
```
impl<T: Into<TokenStream>> iter::FromIterator<T> for TokenStream
```
If given an `IntoIterator<Item = TokenTree>`, it will convert each individual
`TokenTree` to a `TokenStream` (at the cost of two allocations: a `Vec`
and an `Lrc`). It will then merge those `TokenStream`s into a single
`TokenStream`. This is inefficient.

This commit changes the impl to this less general one:
```
impl iter::FromIterator<TokenTree> for TokenStream
```
It collects the `TokenTree`s into a single `Vec` first and then converts that
to a `TokenStream` by wrapping it in a single `Lrc`. The previous generality
was unnecessary; no other code needs changing.

This change speeds up several benchmarks by up to 4%.
    @nnethercote
    nnethercote committed Oct 18, 2019
    Configuration menu
    Copy the full SHA
    a6eef29 View commit details
    Browse the repository at this point in the history

  3. Change Lit::tokens() to Lit::token_tree(). 

    Because most of the call sites have an easier time working with a
`TokenTree` instead of a `TokenStream`.
    @nnethercote
    nnethercote committed Oct 18, 2019
    Configuration menu
    Copy the full SHA
    212ae58 View commit details
    Browse the repository at this point in the history

  4. Change MetaItem::tokens() to MetaItem::token_trees_and_joints(). 

    Likewise for `NestedMetaItem::tokens()`. Also, add
`MetaItemKind::token_trees_and_joints()`, which `MetaItemKind::tokens()`
now calls.

This avoids some unnecessary `TokenTree` to `TokenStream` conversions,
and removes the need for the clumsy
`TokenStream::append_to_tree_and_joint_vec()`.
    @nnethercote
    nnethercote committed Oct 18, 2019
    Configuration menu
    Copy the full SHA
    e4ec4a6 View commit details
    Browse the repository at this point in the history