Reduce size of syntax::tokenstream::TokenTree

[dtolnay]

#53902 raised the size of TokenTree from 32 bytes to 40 bytes, resulting in increased max-rss across a variety of benchmarks by up to 6%.

Here are the data structures involved: playground

Even among the 32 bytes a lot of those were padding. It should be possible to rearrange a few things to drop back to 32 bytes while retaining all the information currently represented.

@nnethercote

[dtolnay]

It may be unfortunate to do this but inlining all the fields of Delimited into TokenTree is able to pack things more efficiently and fit in 32 bytes.

  enum TokenTree {
      Token(Span, Token),
-     Delimited(DelimSpan, Delimited),
+     Delimited(DelimSpan, DelimToken, ThinTokenStream),
  }
-
- struct Delimited {
-     delim: DelimToken,
-     tts: ThinTokenStream,
- }

[eddyb]

@dtolnay Were there APIs on Delimited? I think that's a reasonable inline.
Would it also be possible to move DelimSpan inside Delimited? Or would that not help?

[dtolnay]

Inlining DelimSpan does not help.

There are five methods on Delimited:

rust/src/libsyntax/tokenstream.rs

Lines 46 to 81 in a072d1b

	impl Delimited {
	/// Returns the opening delimiter as a token.
	pub fn open_token(&self) -> token::Token {
	token::OpenDelim(self.delim)
	}
	/// Returns the closing delimiter as a token.
	pub fn close_token(&self) -> token::Token {
	token::CloseDelim(self.delim)
	}
	/// Returns the opening delimiter as a token tree.
	pub fn open_tt(&self, span: Span) -> TokenTree {
	let open_span = if span.is_dummy() {
	span
	} else {
	span.with_hi(span.lo() + BytePos(self.delim.len() as u32))
	};
	TokenTree::Token(open_span, self.open_token())
	}
	/// Returns the closing delimiter as a token tree.
	pub fn close_tt(&self, span: Span) -> TokenTree {
	let close_span = if span.is_dummy() {
	span
	} else {
	span.with_lo(span.hi() - BytePos(self.delim.len() as u32))
	};
	TokenTree::Token(close_span, self.close_token())
	}
	/// Returns the token trees inside the delimiters.
	pub fn stream(&self) -> TokenStream {
	self.tts.clone().into()
	}
	}

[eddyb]

How does DelimSpan compare to what {open,close}_tt compute?
Also, {open,close}_token could be methods on DelimToken itself.
So overall, losing Delimited doesn't feel too bad.

[dtolnay]

The {open,close}_tt methods are passed in a Span. DelimSpan is where the span passed in comes from.

[whitfin]

@dtolnay I was going to take a look at this but it seems it might have already been done (and so this can be closed). The current TokenTree::Delimited looks like this:

pub enum TokenTree {
    Token(Span, token::Token),
    Delimited(DelimSpan, DelimToken, TokenStream),
}

[dtolnay]

Thanks, looks like this was fixed in #56369.