Exception on calling .parse[Source]

[johannegger]

We ran into an exception when calling .parse[Source] on a java.nio.File. Probably this should not happen since .parse returns a Parsed and one therefore doesn't expect the code to throw exceptions.

org.scalameta.invariants.InvariantFailedException: invariant failed:
when verifying ScalametaTokenizer.this.input.chars.apply(curr.endOffset.+(1)).==('\"')
found that ScalametaTokenizer.this.input.chars.apply(curr.endOffset.+(1)) is not equal to '\"'
where ScalametaTokenizer = scala.meta.internal.tokenizers.ScalametaTokenizer@334ebcaa
    at org.scalameta.invariants.InvariantFailedException$.raise(Exceptions.scala:15)
    at scala.meta.internal.tokenizers.ScalametaTokenizer.emitContents$1(ScalametaTokenizer.scala:222)
    at scala.meta.internal.tokenizers.ScalametaTokenizer.loop$1(ScalametaTokenizer.scala:231)
    at scala.meta.internal.tokenizers.ScalametaTokenizer.uncachedTokenize(ScalametaTokenizer.scala:246)
    at scala.meta.internal.tokenizers.ScalametaTokenizer.tokenize(ScalametaTokenizer.scala:34)
    at scala.meta.internal.tokenizers.ScalametaTokenizer$$anon$2.apply(ScalametaTokenizer.scala:262)
    at scala.meta.tokenizers.Api$XtensionTokenizeDialectInput.tokenize(Api.scala:23)
    at scala.meta.tokenizers.Api$XtensionTokenizeInputLike.tokenize(Api.scala:12)
    at scala.meta.internal.parsers.ScalametaParser.scannerTokens$lzycompute(ScalametaParser.scala:204)
    at scala.meta.internal.parsers.ScalametaParser.scannerTokens(ScalametaParser.scala:204)
    at scala.meta.internal.parsers.ScalametaParser.<init>(ScalametaParser.scala:165)
    at scala.meta.internal.parsers.ScalametaParser$$anon$190.apply(ScalametaParser.scala:3596)
    at scala.meta.parsers.Api$XtensionParseDialectInput.parse(Api.scala:21)
    at scala.meta.parsers.Api$XtensionParseInputLike.parse(Api.scala:10)

[xeno-by]

This shouldn't be. How do you invoke parse? Could you show the full stack trace?

[johannegger]

the invocation of parse is here - it's called on a java.nio.file.File
Now it is protected with a Try but that was just introduced to fix the issue.

I will try to get my hands on the full stack trace but it might be difficult - we just saw it once in the logs of a production server.

[xeno-by]

@johannegger Could you provide a problematic file?

[piton4k]

Hi I'm getting the same error, and have been for a while, until I decided to investigate a bit.
Apparently it's caused by any interpolated string (at least any I tried) immediately followed by newline, guess what, with CRLF line end encoding (yay Windows).

Just try this one with CRLF:

object InterpolationError {
  val parse = s"nope"
}

I guess one of those ±1 is not working well with 2 Chars for line end.

Inserting other whitespace characters or line comment after the interpolated string worked to avoid the exception. Or setting the line ends to LF (I know the technology is not there yet), which is a bit difficult on Windows.

[olafurpg]

A quick grep in the parser shows we test against \n in several places

$ rg "\[LF\]"
scalameta/parsers/shared/src/main/scala/scala/meta/internal/parsers/ScalametaParser.scala
257:          if (scannerTokens(i).is[LF] || scannerTokens(i).is[FF]) {
636:      token.is[LF] || token.is[LFLF] || token.is[EOF]
654:      token.is[Semicolon] || token.is[LF] || token.is[LFLF] || token.is[EOF]
697:      token.is[LF] || token.is[LFLF] || token.is[CR]
1535:    if (token.is[LF]) next()
1539:    if (token.is[LF] || token.is[LFLF])
1545:    if (token.is[LF] && ahead { token.is[T] }) newLineOpt()
1550:    if (token.is[LF] && ahead { p(token) }) newLineOpt()

Changing those to .is[Newline] where Newline is a @classifier like here that handles CRLF and LF would probably go a long way to fix these issues. We recently enabled Appveyor CI so we can catch regressions/issues on Windows. Note that we only enable a subset of tests in appveyor.yml, please update it if you add new tests. Would someone with a Windows computer be interested in contributing a fix for this?

[olafurpg]

I spent some time trying to track down this issue in olafurpg@42591c1
The problem appears to be the offsets emitted by LegacyScanner and not in ScalametaParser or ScalametaTokenizer. I don't have more time to investigate right now so I propose we leave this for milestone to not block the upcoming v2 release.

[olafurpg]

The workaround for users affected by this issue is to convert CRLF to LF before running the parser.

[tgodzik]

I can no longer reproduce it on Windows.