Advice on how to do lookahead using the package lexer

[vinipsmaker]

I'm writing a series of Pratt parsing blog posts, and I'm using Rhombus for all code samples. Later down the series I want to teach how to do some advanced stuff such modal lexing + two-phase parsing so one phase uses a cheap greedy lexer definition (modal lexing) to skip over large chunks of code structure just to collect names on global scope (for lexical scoping) and the second phase actually parses the definitions.

I've implemented all this in C++ and I know how to do this using Pratt. I'm not asking for advice on Pratt.

Also I plan on writing a whole series on Pratt parsing. The point of some articles is not just solving the immediate problem, but teaching a tool that you can use later. I want to use lookahead in this example so the reader learns the lookahead trick (other articles in the series are going to use different tricks).

So what I'd like help with is advice on how to use lookahead in the package for the Rhombus lexer. The lexer relies on Port.Input for buffered read (and optional line/column count) which already works great for modal lexing (I can interleave different lexers as I read tokens), but I don't see how I can create a copy of a Port.Input.String in its current state so I can lex the copy to lookahead w/o modifying the original. Is there a method in Port.Input.String that I'm missing? Is there a different approach I should be pursuing for lookahead?

[mflatt]

At the Racket level, I have sometimes use peeking-input-port for this purpose — to get a port that pass to a lexer, but that will not remove input from an original port. There's not yet a Rhombus wrapper for peeking-input-port, but using the Racket binding should work for now.

#lang rhombus
import:
  lib("racket/port.rkt").#{peeking-input-port}

def s = Port.Input.open_string("apple")

s.read_char()

def s2 = (#{peeking-input-port}(s) :: Port.Input)

s2.read_char()
s2.read_char()
s2.read_char()

s.read_char()
s.read_char()
s.read_char()

[mflatt]

Follow-up: rhombus/port now provides port.input.peeking, which is #{peeking-input-port}. I imagine adding a note to the lexer docs, but I have not yet done that.