alex 3.2.7 fails to parse definitions which can pass alex 3.2.6

[Commelina]

I tried the latest alex 3.2.7 on a common-used library language-c-0.9.0.1. However, it failed to parse Lex.x:

src/Language/C/Parser/Lexer.x:123:22: parse error

It seems that the problem happened here:

@iec60559suffix = (32|64|128)[x]?

Then I rolled my alex back to version 3.2.6 then everything worked. I am not sure if it is a bug.

[hasufell]

Same here: https://github.com/haskell/ghcup-metadata/runs/4898784174?check_suite_focus=true#step:8:35

[Ericson2314]

OK I deprecated 3.2.7 and made it unbuildable. We'll need some new test cases for this.

[Ericson2314]

@andreasabel Assuming this isn't a crazy bootstrapping issue, it looks like you made all the parser changes between 3.2.6 and 3.2.7. Would have any time to help debug this?

[andreasabel]

@Ericson2314 : I started to have a look. Then I hit #195.
(I have some old repo of alex which still has a copy of data/AlexTemplate-ghc, so this does not stop me now.)

[andreasabel]

Shrunk example: Issue197.x

{}
@iec60559suffix = (32|64|128)[x]?

tokens :-

$white+         ;
@iec60559suffix ;
{}

Phew, finally found the magic incantations to make bisection work:

$ cat > issue197.sh
cp data/AlexTemplate-ghc data/AlexTemplate.hs
cabal run alex -- Issue197.x
^D
$ chmod +x issue197.sh
$ git bisect start 3.2.7 3.2.6
$ git bisect run ./issue197.sh

Result: d6653f3 is the first bad commit
Author: Andreas Abel
Date: Fri Jan 31 19:09:42 2020 +0100

[ fixed #141 ] regex: allow arbitary repetitions

Previously, the r{n,m} and related forms were restricted to single
digit numbers n and m.

We fix this by recognizing numbers of > 1 digits as NUM token and adding
rules to the regex parsing that also allows NUMs for n and m.
Previously, only CHAR was allowed, which subsumes single digits.

[Ericson2314]

Running info on the happy grammar,

...
  58     rep -> '{' CHAR '}'                                (49)
  59     rep -> '{' CHAR ',' '}'                            (50)
  60     rep -> '{' CHAR ',' CHAR '}'                       (51)
  61     rep -> '{' NUM '}'                                 (52)
  62     rep -> '{' NUM ',' '}'                             (53)
  63     rep -> '{' NUM ',' NUM '}'                         (54)
...
  71     set0 -> CHAR                                       (62)
  72     set0 -> CHAR '-' CHAR                              (63)
...

The problem is these lower rules don't have NUM equivalents.

I think remembering to double up NUM and CHAR is hard, and all the more so when this case we would need to "unparse" the number, yet also preserve things like 001 vs 1.

This makes makes we think we should not have a NumT, and instead just have a number nonterminal, parsed from char terminals. Does that sound good?

[andreasabel]

Er, @Ericson2314, I am already working on this. I just lack the repo rights to assign this to me so that you would know about it.

[andreasabel]

This makes makes we think we should not have a NumT, and instead just have a number nonterminal, parsed from char terminals. Does that sound good?

Yes this sounds good.
I just wish you would not ask me to help and then race for a fix. Wasted my time, I could have looked into something else.