Add lexer regression tests to the parser crate

[adampetro]

Why

We will upgrade the logos crate soon. The lexer gets untrusted input. A change in logos behavior can cause incorrect tokens, incorrect spans, out-of-bounds data access, or a denial of service. These tests pin the current behavior and make regressions visible.

This change adds tests only. It does not change production code.

What

Unit tests in src/lexer/logos_lexer.rs:

• Test all punctuators, variable names, and ignored tokens, with exact spans.
• Test edge cases in strings, block strings, and numbers.
• Compare the full token stream for a document with all token types and all ignored token types.
• Test the spans of tokens that come after strings. String parsing moves the outer lexer with remainder() and bump(). These logos APIs are the most likely place for subtle changes between versions.
• Test the LogosLexer iterator, the token count, and the max tokens limit.

Safety tests in src/lexer/logos_lexer/safety_tests.rs:

• Lex a corpus of hostile inputs. Assert that the lexer terminates, makes progress, consumes the full input, and returns spans that are in bounds and on char boundaries. Spans with incorrect bounds can cause panics in error formatting, which slices the source by span.
• Lex each prefix and each suffix of a document. This simulates unexpected end of input in each lexer state.
• Lex large hostile inputs that must complete in linear-like time.
• Pin the current stack usage for long numeric tokens and long runs of ignored characters.

Integration tests in tests/lexer_adversarial_test.rs:

• Parse hostile documents through the public API and convert the errors to GraphQL errors.
• Show that the max tokens limit bounds the work for large documents.

Known issue found while writing these tests

With logos 0.15, the generated matchers for numeric tokens and for runs of ignored characters use stack space proportional to the run length in unoptimized builds. Optimized builds compile the recursion into loops and are not affected. As a result of this, one numeric token with approximately 5,000 or more digits, or one run of approximately 50,000 or more ignored characters, causes a stack overflow in unoptimized builds. The max tokens limit does not protect against this, because each input is a single token.

Two tests document this issue. They are marked with #[ignore] because a failure aborts the test process. After each logos upgrade, run this command in a debug build:

cargo test -p bluejay-parser -- --ignored

If the tests pass, remove the #[ignore] attributes.

How to verify

cargo test -p bluejay-parser --features format-errors

[swalkinshaw]

Good idea 🚀