Change parser mode and state with a token output

[TheOnlyTails]

I'm building a lexer, and I want it to, given a closing brace (}), produce a different token depending on the current parser state, and then change the state for the next token. I've looked at the docs a bunch but I can't find the exact way to do what I want, and I'd appreciate some help.

In this specific case, the state is this struct:

struct LexerState {
	/// how deeply nested the current code is, increased when encountering #{ and { tokens, decreased for }
	nesting: u32,
	/// either Normal or String
	mode: LexerMode,
}

[zesterer]

So first of all, I think you're framing this in an unhelpful way in your head. Chumsky is a declarative API: it's specifically designed to avoid needing to think about parser 'state' in this way.

To implement this, you'll probably want to look into the context-sensitive parsers, such as with_ctx. You'll want to start off with your top-level parser passing a context of 0 (the initial nesting level) and then incrementing that by 1 for each level of nesting.

[TheOnlyTails]

@zesterer I'm mostly having a hard trouble understanding how to increment and decrement the nesting level when encountering specific tokens. I assume it has something to do with a combination of configure and map_ctx, but I can't seem to figure it out.

[zesterer]

There should be no need to decrement: the context-sensitive combinators are scoped, so you only need to increment on each level of recursion. You can do that with the map_ctx combinator. configure shouldn't actually be required here, it's used to configure the behaviour of a parser: but in this case, you don't want to change the behaviour of the parser, only provide the context to your AST. When using map_with to create your AST node, the extra parameter gives you access to the current context.

Here's a rough sketch of what this might look like. Note that I've not checked that this compiles, so there will likely be things to fix and things you want to adapt for your specific needs. Hopefully it gives some idea of the broad approach though.

enum Token {
    Braces { nesting: u32, children: Vec<Self> },
    ...
}

let token = recursive(|token| {
    let braced = 
        // Here, we increment the nesting level for inner tokens
        token.map_ctx(|nesting| nesting + 1)
        .repeated().collect()
        // When creating the braced AST node, we use the nesting level context
        .map_with(|children, extra| Token::Braces {
            nesting: extra.ctx(),
            children,
        })
        .delimited_by(l_brace, r_brace);

    choice((
        braced,
        ... // Other tokens
    ))
})
    // The starting nesting level is 0: this gets incremented at each level of recursion using `map_ctx` above
    .with_ctx(0);