Escapes are broken

[jiribenes]

Currently, unicode escapes themselves are valid characters which is somewhat confusing, I'd expect the following escape to have to be written as '\u0a00' instead of \u0a00:

I'd propose only allowing unicode escapes inside of string or char literals.

Here's the relevant code:

effekt/effekt/shared/src/main/scala/effekt/Lexer.scala

Line 584 in d98362e

case '\\' if nextMatches("u") => matchUnicodeLiteral()

effekt/effekt/shared/src/main/scala/effekt/Lexer.scala

Lines 448 to 469 in d98362e

	/** Matches a character: '.+' */
	def matchChar(): TokenKind = {
	if (peekMatches(_ == '\'')) err("Empty character literal.")
	consumeWhile(_ != '\'')
	expect('\'')
	// exclude opening and closing '
	val cs = slice(start + 1, current - 1)
	TokenKind.Chr(cs.codePointAt(0))
	}
	lazy val hexadecimal = ('a' to 'f') ++ ('A' to 'F') ++ ('0' to '9')
	/** \u<HEXADECIMAL>+ */
	def matchUnicodeLiteral(): TokenKind = {
	consumeWhile(hexadecimal.contains(_))
	val n = slice(start + 2, current)
	try {
	TokenKind.Chr(Integer.parseInt(n, 16))
	} catch {
	case e: Throwable => err("Invalid unicode literal.")
	}
	}

---

Related to this: we allow wildly nonstandard escapes in string literals ("anything goes"):

effekt/effekt/shared/src/main/scala/effekt/Lexer.scala

Lines 408 to 414 in d98362e

	case Some('\\') => {
	expect('\\')
	peek() match {
	case Some('\"') | Some('\\') | Some('n') | Some('t') | Some('r') => consume()
	case Some('u') => {
	consume()
	consumeWhile(c => !c.isWhitespace && c != '"')

This requires some design, but I'd prefer to agree on a common format of an escape, for example Zig has:
\u{NNNNNN} as "hexadecimal Unicode scalar value UTF-8 encoded (1 or more digits)"
I like it because it allows multiple digits (subsuming the need for separate \xNN and \uNNNN and \uNNNNNN and whatever else) and it is easy to parse (<escape> ::= '\u{' <hex>+ '}')

I think it's also fine to invest a little bit into lexing escapes properly by ourselves :)

[jiribenes]

Related:

effekt> '\n' == '\t'
true

cc @marzipankaiser

Let's just please fix the lexer for strings and characters incl. escapes. 🙏

[jiribenes]

Another issue: val c = '\' could have two possible meanings:

1. a character that is not completely parsed yet that contains an escaped ' inside
2. a character that completely parsed and contains just \ inside

Currently, we choose 2, but it’s inconsistent with what I presumed when making the highlighting (1 — what Scala does), and with common expectations of programmers & LLMs.
We agreed on Slack to ban this as an unclosed character and require writing val backslash = '\\' for a backslash and val apostrophe = '\'' for an apostrophe.

[marzipankaiser]

Another issue: val c = '\' could have two possible meanings:

1. a character that is not completely parsed yet that contains an escaped ' inside
2. a character that completely parsed and contains just \ inside

Currently, we choose 2, but it’s inconsistent with what I presumed when making the highlighting (1 — what Scala does), and with common expectations of programmers & LLMs. We agreed on Slack to ban this as an unclosed character and require writing val backslash = '\\' for a backslash and val apostrophe = '\'' for an apostrophe.

In #622, I changed it to 1.