Explicit pre/post skip

[mavam]

I'd like to continue with the discussion of #46 in slightly broader context.

IIUC, lexy has a concept of "whitespace" that can applied in a post-skip manner, i.e., after the production. I was wondering a few things:

1. Why call it "whitespace" when it can be in theory any parser that skips the input? Coming from Spirit, I find it more natural to think in terms of "skipping" because based on what you do, it's not just whitespace.

2. Would it make sense to make the skipping configurable, by having a pre and post skip mode? Basically, I was thinking of something along the lines of this:

   struct config {
     static constexpr auto pre_skip = /* parser */;
     static constexpr auto post_skip = dsl::whitespace;
   };

3. This is mostly a question of understanding: is inheriting from lexy::token_production the same as Spirit's lexeme directive?

[foonathan]

1. Why call it "whitespace" when it can be in theory any parser that skips the input? Coming from Spirit, I find it more natural to think in terms of "skipping" because based on what you do, it's not just whitespace.

It is designed for specifically skipping whitespace. In particular, it is optimized to skip sequences of ASCII space, etc. It's also the main use case, I think?

2. Would it make sense to make the skipping configurable, by having a pre and post skip mode?

I'm not sure whether it would make sense: why do you need to skip different things before and after? Do you have a use case? It's also problematic to implement: Consider a >> b | c >> d. Currently, lexy attempts to match a or c to determine which branch to take. If a/c have a (potentially distinct!) pre whitespace skipper, that's more complicated.

3. This is mostly a question of understanding: is inheriting from lexy::token_production the same as Spirit's lexeme directive?

I haven't actually used Spirit, but I think so. It disables all whitespace skipping within the production (unless there is an explicit whitespace directive), but instead skips whtiespace once after.

[mavam]

It is designed for specifically skipping whitespace. In particular, it is optimized to skip sequences of ASCII space, etc. It's also the main use case, I think?

Got it. Coming from spirit, the skipper is also just a regular parser, which is why asked.

I'm not sure whether it would make sense: why do you need to skip different things before and after?

Not necessarily different things, it could be the same skip parser. But having the ability to say I skip whitespace before or after the respective parser is a neat option in Spirit.

Do you have a use case? It's also problematic to implement: Consider a >> b | c >> d. Currently, lexy attempts to match a or c to determine which branch to take. If a/c have a (potentially distinct!) pre whitespace skipper, that's more complicated.

On the implementation side, I would imagine it's just x >> s or s >> x for a skipper s and parser x. It would then be transparent to the choice parser.

I haven't actually used Spirit, but I think so. It disables all whitespace skipping within the production (unless there is an explicit whitespace directive), but instead skips whtiespace once after.

👍

[foonathan]

Not necessarily different things, it could be the same skip parser. But having the ability to say I skip whitespace before or after the respective parser is a neat option in Spirit.

If you have an actual use case, I can reconsider adding it.

On the implementation side, I would imagine it's just x >> s or s >> x for a skipper s and parser x. It would then be transparent to the choice parser.

Note that the way the parser works, it will only use the part before >> as choice condition and never backtrack past it. So if you have skipper s and rules a and b with pre-skipping the choice a | b would actually be a >> s | s >> a + s | b >> s | s >> b + s, where the second one never matches. A correct transformation would be a >> s | s >> (a >> s | b >> s) | b >> s, but that's not trivial to produce.