noob Q: parsimonious does not consume (skip) the white spaces by default?

[mw66]

Hi, I just tried this modified example from the readme:

from parsimonious.grammar import Grammar

grammar = Grammar(
    """
    bold_text  = bold_open text bold_close
    text       = ~"[A-Z0-9_]+"i   # remove the space from regex, and at least 1 char, +
    bold_open  = "(("
    bold_close = "))"
    """)

print(grammar.parse('(( bold_stuff ))'))  # used _

I'm expecting "bold_stuff" be parsed as text; but I got this error:

parsimonious.exceptions.ParseError: Rule 'text' didn't match at ' bold_stuff ))' (line 1, column 3).

in most programming languages, using white spaces as separator between different syntactic elements does not need special attention, so do I have to write rule like this:

    bold_text  = bold_open white_spaces text white_spaces bold_close
    white_spaces = ...

Then this is too tedious for any serious programming language grammar definition, and nobody care about the white_spaces even if it's parsed.

Is there a parser flag somewhere to let the parser consume / skip white spaces by default?

Am I missing something?

[mw66]

OK, I found #118

And the workaround there:

Define white space as _, and

"Hang it off every leaf node of your grammar" ??? !!!

Am I the only one who think this idiom is tedious and ugly?

[lucaswiman]

Is there a parser flag somewhere to let the parser consume / skip white spaces by default?

No.

in most programming languages, using white spaces as separator between different syntactic elements does not need special attention, so do I have to write rule like this:

It actually does need special attention when parsing those programming languages. There's usually a separate "lexing" phase that turns the program into a series of tokens. That will usually abstract away whitespace, though I think it's done more for efficiency than convenience. An LL(1) parser is much more powerful and faster if it can look one token ahead instead of one character ahead.

Parsimonious combines lexing and parsing-of-tokens into a single operation (parsing), which makes the parse trees a bit uglier.

Am I the only one who think this idiom is tedious and ugly?

No, though it's also not that big of a deal in practice. Usually you don't operate on parse trees directly: you can write a few methods in your NodeVisitor subclass to not care about nodes you don't care about. See e.g. #114 and some linked issues for some more discussion.

If you'd prefer a more traditional lexing approach, Lark has somewhat similar APIs to Parsimonious, at least for grammar definition, but directly supports lexing and has a syntax for ignoring nodes in the parse tree. Lark also has a bunch of extremely cool features (like integration with hypothesis), so it's worth taking a look at.

The disadvantages of Lark are also fairly clear. CFGs can sometimes be more annoying than PEGs due to ambiguity. And unless your grammar works with an LALR(1) parser, it may be less efficient than parsimonious.

[mw66]

"""
Am I the only one who think this idiom is tedious and ugly?

No, though it's also not that big of a deal in practice. Usually you don't operate on parse trees directly:...
"""

You misunderstood me: I know it won't affect the visitor code, but it makes the grammar definition file ugly: so many useless_ scattering around.

Also, this makes the grammar file locked to parsimonious, when user want to use another PEG parser, the grammar files need to be changed, that is another issue I raised in #191

[mw66]

FYI:

https://github.com/PhilippeSigaud/Pegged/wiki

By default, the grammars do not silently consume spaces, as this is the standard behaviour for PEGs. There is an opt-out though, with the simple < arrow instead of <- (you can see it in the previous example).

I think this < extension is very convenient.