Feature: allow Option<Skip>, Result<Filter<T>, _> in callbacks

[Jeremy-Stafford]

Currently, logos does not allow nesting Skip and Filter in Option or Result.

This makes the following pattern very difficult (impossible?) to express:

• Match a "start" pattern.
• Compute the number of characters to skip, raising an error if this is not possible.
• Match a "end" pattern.
• Skip the whole token.

This pattern appears when lexing C-style block comments (/* ... */). For languages which forbid nesting (including C itself), this can be done using a regex (read: unreadable mess of stars and slashes). For languages which allow nesting, this cannot be fixed.

The current callback-result system is implemented as a trait. It would therefore be impossible to allow Option<Skip> (that would introduce overlapping implementations). This feature would therefore require a type defined as

enum FilterResult<T> {
    Error,
    Skip,
    Ok(T),
}

[evbo]

@Jeremy-Stafford thank you and the contribution you made to the docs is excellent! Works like a charm :)