How to have variable grammar?

[mliebelt]

I have thought the last view weeks over the options to have a PGN parser that fits to all wishes, and I think peg.js (or its successor peggy) will not solve it. I did some investigation, and there are some links:

• Parametrize the grammar by externally-supplied variables pegjs/pegjs#36 Have external configuration that parametrizes the results of a grammar rule or even the grammar rule itself.
• Implement parametrizable rules pegjs/pegjs#45 Similar ticket, with similar ideas, and no implementation yet.

Problem to solve?

Someone has a database of pgn files, with the following requirements:

• She wants to read just the moves, and ignore the tags all together, even when the tags will contain errors.
• She wants to read all games, but only keep the seven roster tags, not the other ones.
• She wants to read all games, and record the wrong tags, but continue parsing all games.
• She wants to read all games, and stop parsing then the first rule breaks.

Now some other setup, other requirements:

• She wants to read all games, and order them by date. Therefore only valid date games are possible.
• She wants to read all games, and save them later in junks by year. So the year should be readable, or be in the format as expected by the specification. (10 characters, with 8 digits | question mark like 1992.??.??, 192?.??.??, 1992.12.??, 1992.12.24, ...)
• She wants to read all games, save the ones with correct Date tag in separate junks, and all others then in the rest junk.

And these are only 2 of many possibilities, that could be requirements by someone.

Possible solutions

Here are some ideas what could be options in having that possibilities:

• Rules should be robust enough to have a catch all at the end, if that is a reasonable result. So ignore not well-formed tags if they are at least strings. Read all moves in any format, as long as they have ranks and files that go from a-h and 1-8 only. ...
• Allow different grammars, by having parts of the grammar build as variants. Depending on the requirements, that would allow
  • tags according to the spec only
  • tags with some variants (upper-/lowercase), but correct values
  • tags with string values only
  • tags even with any additional tags
  • ...
    Similar to the other parts of the spec. You can then mix and match different variants, and for each variant build a grammar, and generate a parser for it. At the end, we could have not one, but 20 parsers built.
• Have additional functionality (not parser rule based), to fix after parsing some deficits. So parsing should work most of the time, but a post-parsing step could then produce warnings or errors depending on the options given. See Ignores FEN turn -> Always sets first move turn as "w" #62 for an example when parsing is not sufficient ... Decide if having some state during parsing (turn from FEN) is worth it ...

[mliebelt]

One idea I had:

• Have a grammar that is robust and reads (nearly) everything.
• When reading many games, split them into game chunks (should be easy enough without a grammar), and read all games one by one. If the parser throws an exception, that can be caught, and recorded with the (not successful read) game.
• Differentiate strict and not so strict rules
  • Strict rules are used by default.
  • Not so strict rules will record additional "warnings", which are collected.
  • A validator may differentiate depending on properties of the configuration.

So the following should be possible:

1. Have a validator / post processor for strict games
   • Config { mode: strict }
   • Result: list of games, games without warnings are ok, games with warning were read, but have non-strict rules used, games with errors are recorded to have something to check for modification.
2. Have a validator / post processor for games notation only, that filters all tags (including the warnings).
3. Have a validator / post processor for reading only the 4 roster tags, and filtering all other.

[MichaelLeeHobbs]

Take a look at https://microsoft.github.io/monaco-editor/. It more than a parser though so likely doesn't fit the bill.

[MichaelLeeHobbs]

Here's another requirement:
She wants to read all games from multiple files and identify duplicates where there are possible formatting differences.

[mliebelt]

That is a good one. So in the future, it would be possible to have a bunch of files e.g. in a directory, and the parser reads all, collects information about it, and reports that. Nice one, will generate an issue from that.