parsing fails with apparently valid grammar

[dcourtois]

Hi,

I discovered this project a few days ago, and I've been playing with it in my spare time, but I encountered what I think might be a bug. I write "think" because I'm pretty new to this field, so I might be doing something obviously wrong (in which case I apologize in advance for wasting your time :p)

Here is a really simple test:

const std::string tests [] = {
    "1.2",      ".2",       "1.",       "1",
    "1.2e-10",  ".2e-10",   "1.e-10",   "1e-10",
    "1.2e+10",  ".2e+10",   "1.e+10",   "1e+10",
    "1.2e10",   ".2e10",    "1.e10",    "1e10",
    "[1.2]",    "[1.]",     "[.2]",     "[1]"
};

namespace grammar
{
    using namespace tao::pegtl;
    using namespace tao::pegtl::ascii;

    // exponent : [Ee][+-]?\d+
    struct _expo : seq< one< 'e', 'E' >, opt< one< '+', '-' > >, plus< digit > > {};
    
    // number : \d+{_expo}? | \d*\.\d+{_expo}? | \d+\.\d*{_expo}?
    struct _real : sor<
        seq< plus< digit >, opt< _expo > >,
        seq< star< digit >, one< '.' >, plus< digit >, opt< _expo > >,
        seq< plus< digit >, one< '.' >, star< digit >, opt< _expo > >
    > {};

    // test : {_real} | [{_real}]
    struct _expression : sor<
        _real,
        seq< one< '[' >, _real, one< ']' > >
    > {};

}

int main(int argc, char ** argv)
{
    using tao::pegtl::parse;
    using tao::pegtl::string_input;

    for (const auto & test : tests)
    {
        bool result = parse< grammar::_expression >(string_input< >(test, test));
        std::cout << test.c_str() << " : " << (result ? "ok" : "not matched") << std::endl;
    }

    return 0;
}

This is simplified, but basically it should match either a number or a number enclosed in square brackets. All the first tests are matched correctly (the numbers without brackets) but it fails to match numbers inside squared brackets in some cases. Here is the output:

1.2 : ok
.2 : ok
1. : ok
1 : ok
1.2e-10 : ok
.2e-10 : ok
1.e-10 : ok
1e-10 : ok
1.2e+10 : ok
.2e+10 : ok
1.e+10 : ok
1e+10 : ok
1.2e10 : ok
.2e10 : ok
1.e10 : ok
1e10 : ok
[1.2] : not matched
[1.] : not matched
[.2] : ok
[1] : ok

I tried using the tracer utility to understand what I was doing wrong, but I couldn't find any obvious mistake ... Maybe it has to do with how I create the string_input but I'm not sure how to use it correctly. I don't understand the difference between the data and source parameters as the documentation is not really clear on the subject (well, at least not for me :p)

[d-frey]

There are two problems: You don't have a grammar that matches the whole input and you have the wrong order of choices in _real. The second problem leads to seq<A,B> to not even try to match B when is is able to match A. In your case, seq< plus< digit >, opt< _expo > > matches the 1 of the input 1.2. So it doesn't even look at the other choices. Since your grammar does not require eof at the end, this does match - but not the whole input. When you have [1.2] as an input, it matches [, then matches 1 for _real and then expects ], but gets . - hence it fails on this input.

To solve it, use

struct _expression : seq< sor<
    _real,
    seq< one< '[' >, _real, one< ']' > >
>, eof > {};

to make sure you always match the complete input and use:

struct _real : sor<
    seq< star< digit >, one< '.' >, plus< digit >, opt< _expo > >,
    seq< plus< digit >, one< '.' >, star< digit >, opt< _expo > >,
    seq< plus< digit >, opt< _expo > >
> {};

to always get the longer matches first. Note that this is one of the key differences between a PEG and a CFG.

Feel free to re-open this issue if this doesn't solve your problem.

[dcourtois]

Hi @d-frey ,

Thanks a lot for taking the time to answer, that was fast ! It solves my problem (the order of the _real rule) and I just noticed the matches ... **in the given order** part in the documentation, so it was indeed me doing something obviously dump, sorry about that :)

While I'm at it, is there any place in the documentation where it's made clear what exactly are the roles / differences of the 2 constructors' parameters of string_input (namely data and source) ?

[skyrich62]

Indeed there is. Inputs-And-Parsing in the doc folder explains everything. The first paragraph of that page explains "source" for all of the input methods.

To paraphrase: "data" is the input your are parsing, "source" is where that input comes from. For a file parser, "source" would be the name of the file, for example.

[d-frey]

Yes, data is the actual data to parse while source is just passed on for error reporting/debugging purposes. If you are parsing a file, it is usually the filename, for in-memory parsing it is often useless (and is therefore defaulted).

[dcourtois]

Haha ok, we're just not aligned on the definition of 'clear' :) I had read this page (multiple times even) but the construction of string_input was still confusing me : I couldn't understand why both data and source were needed. I understand the concept of source for files, but wasn't sure what it was for strings (e.g. what happens if data is different from source in string_input's constructor)
Maybe having a simplified constructor with only source would help ?

Anyways, thanks for the explanations !

[skyrich62]

Hi,
I've sometimes found that setting source and data to the same string helps with debugging the grammar when doing tracing. Otherwise, for string_input, I set the source to an empty string.

[robinmoussu]

Hello, I don't want to create a new issue, since I had exactly the same question than dcourtois and you solved it. I was not able to understand what I had to put for data in the constructor of string_input. Whith your explanation, it is clear.

In the documentation, I think you should rephrase

In the context of PEGTL input classes and positions, source is a string that identifies where the to-be-parsed data comes from. For example when parsing a file, the filename is the source.

to something like

In the context of PEGTL input classes and positions, source is a string that identifies where the to-be-parsed data comes from. Data is the actual data to parse while source is just passed on for error reporting/debugging purposes. If you are parsing a file, it is usually the filename, for in-memory parsing it is often useless (and is therefore you can use an empty string).

I just have one additionnal remark, when you said

and is therefore defaulted

What do you mean by that? I if try to create a string_intput<> by only providing one argument (data), I get a compile error.

[ColinH]

Thanks, I'll reopen this issue as a reminder to look at the documentation again.

[ColinH]

The defaulted part is not correct (anymore).

[ColinH]

The role of the source parameter should now be more clear.

[robinmoussu]

In my opinion, it is still really unclear what you should put (based on the documentation, not this discussion) if your source comes from memory or a string. You may consider adding some examples.