Errors at EOF

[shwestrick]

Problem

Error positions are currently reported by attaching to a token, but this doesn't work at the end of the file.

For example, on a file containing just the word val, the parser attempts to parse a pattern immediately after the val, but since this is the last token of the file, it crashes.

This particular example crashes at the call tok i below, because it is an out-of-bounds access.
https://github.com/shwestrick/parse-sml/blob/6e61e1e431ca2aa8f4ddf6f3a47a9d1f8a0b5851/parse/ParsePat.sml#L106-L111

But this problem is more pervasive: many of the errors throughout the parser are similarly designed.

Suggested Solution

Introduce a new error function which similar to previous but now is aware of the token stream, e.g.

fun error (toks: Token.t Seq.t) {pos: int, what: string, explain: string option} =
  if pos >= Seq.length tokens then
    ... (* EOF error *)
  else
    ... (* normal error *)

Then all of the error sites need to be replaced (should be a nice little bit of zen refactoring 💆🏻‍♂️)