Consider exposing the lexer

[dabrahams]

Description

I am able to use the parser to get the tokens, but the tokens are all I really need and building syntactic structure on top of it is wasteful. Not the end of the world for my application, but it would be cleaner if I could just get a token stream.

[ahoppen]

I don’t think you would want to use the tokens out of the lever straight away because the parser may modify the lexemes, eg.

• The parser performs state transitions in the lever during string interpolation parsing

  swift-syntax/Sources/SwiftParser/StringLiterals.swift

  Lines 566 to 576 in ef367f7

  	// The parser has more knowledge that we have reached the end of the
  	// string interpolation now, even if we haven't seen the closing ')'.
  	// For example, consider the following code
  	// "\(abc "
  	// Since the lexer doesn't know anything about the expression structure,
  	// it assumes that the `"` starts a new string literal. But since we
  	// know in the parser that an identifier cannot be followed by a string
  	// literal without a connecting binary operator and can thus consider
  	// it as the surrounding string literal end, which thus also terminates
  	// the string interpolation.
  	self.lexemes.perform(stateTransition: .pop, currentToken: &self.currentToken)

• The parser re-classifies trivia during string interpolation parsing
  

  swift-syntax/Sources/SwiftParser/StringLiterals.swift

  Lines 215 to 217 in ef367f7

  	// The newline at the end of the last line in the string literal is not part of the represented string.
  	// Mark it as trivia.
  	let content = self.reclassifyTrivia(in: lastMiddleSegment.content, trailing: newlineSuffix)

• The parser may decide to only consume the prefix of a lexeme, eg. during parsing of generics
  

  swift-syntax/Sources/SwiftParser/Nominals.swift

  Line 347 in ef367f7

  let langle = self.consumePrefix("<", as: .leftAngle)

[dabrahams]

Interesting. I just need to skip past balanced pairs of {...} to look for the next “}” in my input, but if the parser is needed to deduce where the bounds of a string literal are that would definitely require the parser. Hmm, how does SwiftSyntax express a string literal with interpolations in it? Is the literal still a token?

[ahoppen]

I think matching braces should be doable in the lever alone but the distinction about what the lexer can do by itself vs for what it needs the parser are too subtle for me to be confident to expose the lexer by itself.

swift-syntax parses the string literal into string and interpolation segments: https://swiftpackageindex.com/swiftlang/swift-syntax/main/documentation/swiftsyntax/stringliteralexprsyntax