RFC: On parse error - show the next possible tokens

[da-x]

People asked for a patch upstream - so here's the place to start a discussion.

In this change, I've made sure that 'cabal test' still passes, and the interface to unmodified users is preserved. But you would probably have more comments.

[da-x]

I've added an additional commit to the branch, per your review. Feel free to squash/amend send more comments.

As for your request, this seems to be taking compiler/stage1/build/Parser.o from 2.2MB to 7.5MB. I'm guessing that if happy_explist_per_state was Int -> [ByteString], and with OverloadedStrings, along with unpack around its use perhaps would optimize it better? Or, maybe we can add a huge where clause instead of duplicating string literals?

[simonmar]

300%+ increase in code size to too much (a lot too much!). In Happy-generated code we can use whatever tricks we like, since it's not meant to be user-editable code. For instance, we use unboxed values all over the place. I suggest there should be a single array of token names (each an unboxed string), and an unboxed array, indexed by state, with a bitmap of the allowed token values.

[da-x]

Followed your suggestion - now compiler/stage1/build/Parser.o is 2.4MB. Please see the commit added to the branch.

[da-x]

Just rechecking - is there anything more to be done for this?

[simonmar]

This looks good. I'll find some time to test it and merge.

[da-x]

Rebased against simonmar/happy#master

However, cabal test seems to show some failures even with simonmar/happy#master, under ghc 7.10.2. With da-x/happy#master I get the same failures.

[da-x]

Rebased against simonmar/happy#master. Rebased my ghc changes to use the new happy feature on top of ghc#master.

Original branch simonmar/happy#master still fails cabal test with ghc 7.10.2:

monad002.g.hs:293:22:
    Couldn't match kind ‘*’ with ‘#’
    When matching types
      t0 :: *
      Happy_GHC_Exts.Int# :: #
    Expected type: t0
                   -> t0
                   -> Token
                   -> HappyState Token (t -> [Char] -> Int -> ParseResult a)
                   -> [HappyState Token (t -> [Char] -> Int -> ParseResult a)]
                   -> t
                   -> [Char]
                   -> Int
                   -> ParseResult a
      Actual type: Happy_GHC_Exts.Int#
                   -> Happy_GHC_Exts.Int#
                   -> Token
                   -> HappyState Token (t -> [Char] -> Int -> ParseResult a)
                   -> [HappyState Token (t -> [Char] -> Int -> ParseResult a)]
                   -> t
                   -> [Char]
                   -> Int
                   -> ParseResult a
    Relevant bindings include
      i :: t0 (bound at monad002.g.hs:293:18)
      cont :: t0 -> [Char] -> Int -> ParseResult a
        (bound at monad002.g.hs:293:13)
    In the expression: action i i tk (HappyState action) sts stk
    In an equation for ‘cont’:
        cont i = action i i tk (HappyState action) sts stk
    In the expression:
      let cont i = action i i tk (HappyState action) sts stk
      in
        case tk of {
          TokenEOF -> action 19# 19# tk (HappyState action) sts stk
          TokenLet -> cont 8#
          TokenIn -> cont 9#
          TokenInt happy_dollar_dollar -> cont 10#
          TokenVar happy_dollar_dollar -> cont 11#
          TokenEq -> cont 12#
          TokenPlus -> cont 13#
          TokenMinus -> cont 14#
          TokenTimes -> cont 15#
          TokenDiv -> cont 16#
          TokenOB -> cont 17#
          TokenCB -> cont 18#
          _ -> happyError' tk }

monad002.g.hs:296:26:
    Couldn't match kind ‘*’ with ‘#’
    When matching types
      t0 :: *
      Happy_GHC_Exts.Int# :: #
    Relevant bindings include
      cont :: t0 -> [Char] -> Int -> ParseResult a
        (bound at monad002.g.hs:293:13)
    In the first argument of ‘cont’, namely ‘8#’
    In the expression: cont 8#
    In a case alternative: TokenLet -> cont 8#

[sinelaw]

+1

Is there anything missing for this to be merged?

[simonmar]

I got test failures, e.g.

../dist/build/happy/happy --strict --template=.. -g monad002.ly -o monad002.g.hs
ghc -hide-all-packages -package base -package array -package mtl -Werror -fforce-recomp  monad002.g.hs -o monad002.g.bin
[1 of 1] Compiling Main             ( monad002.g.hs, monad002.g.o )

monad002.g.hs:311:22:
    Couldn't match kind ‘*’ with ‘#’
    When matching types
      t0 :: *
      Happy_GHC_Exts.Int# :: #

(edited, had the wrong error message)

[da-x]

Seem related to something else - you get the same errors without this pull request, using yesterday's 1.19.6 tag, due to GHC 7.10.

With GHC 7.8.4 the tests pass (and separately, also with the changes here).

[da-x]

Opened a separate pull request for that test error.

[simonmar]

This broke the Travis build, could you take a look please? https://travis-ci.org/simonmar/happy/builds/133044562

[da-x]

Looking into this

[da-x]

Fixed in a new pull request #68