Issue with the io Lexer

[bivab]

I found an issue when passing the string "()", "[]" or "{}" to Compiler tokensForString. The first token returned has its line slot set to 0.
When i pass any other string, such as "Foo bar()", the first token has the line slot set to one.

The following command should return true
Compiler tokensForString("()") at(0) line == Compiler tokensForString("a") at(0) line

[dennisferron]

A while back when I was profiling Io using valgrind + kcachegrind, that tool found a reference to uninitialized memory in Lexer.c. Maybe the line = 0 you are seeing is uninitialized memory? I didn't have time to investigate it further at the time.

[stevedekorte]

Fixed. When the input begins with (, [, or {, the lexer inserts a synthetic group-name identifier token before consuming the opening delimiter. At that moment current is still sitting at the start-of-input sentinel (charLineIndex[0]), which caused the <= walk in IoLexer_currentLineNumber to return 0. Lines are 1-based, so now clamp a 0 result to 1.

Io> Compiler tokensForString("()") at(0) line
==> 1
Io> Compiler tokensForString("[]") at(0) line
==> 1
Io> Compiler tokensForString("{}") at(0) line
==> 1

Commits:

• wasm branch: 0895e6e
• master branch: 5d3899b

Closing.