Skip to content

Commit

Permalink
fix: properly set location for string tokens ending in a newline
Browse files Browse the repository at this point in the history
Fixes decaffeinate/decaffeinate#457

The existing logic for computing the end location of a string was to take the
end of the string contents, then add the delimiter length to last_column. For
example, `"""abc"""` would have an end position three characters after the `c`.
However, if a string ended in a newline, then the end location for the string
contents would be one line above the end location for the string, so the proper
fix is to move the end location to the next line, not just to shift it to the
right.

This avoids a bug where the location data would sometimes reference a
non-existent location (one past the end of its line), which would confuse
decaffeinate-parser.
  • Loading branch information
alangpierce committed Oct 18, 2016
1 parent f9a8f54 commit b60b9ed
Show file tree
Hide file tree
Showing 3 changed files with 34 additions and 2 deletions.
7 changes: 6 additions & 1 deletion lib/coffee-script/lexer.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 5 additions & 1 deletion src/lexer.coffee
Original file line number Diff line number Diff line change
Expand Up @@ -530,7 +530,11 @@ exports.Lexer = class Lexer

[firstToken, ..., lastToken] = tokens
firstToken[2].first_column -= delimiter.length
lastToken[2].last_column += delimiter.length
if lastToken[1].substr(-1) == '\n'
lastToken[2].last_line += 1
lastToken[2].last_column = delimiter.length - 1
else
lastToken[2].last_column += delimiter.length
lastToken[2].last_column -= 1 if lastToken[1].length is 0

{tokens, index: offsetInChunk + delimiter.length}
Expand Down
23 changes: 23 additions & 0 deletions test/location.coffee
Original file line number Diff line number Diff line change
Expand Up @@ -528,6 +528,29 @@ test "Verify real CALL_END tokens have the right position", ->
eq callEnd[2].first_column, startIndex + 2
eq callEnd[2].last_column, startIndex + 2

test "Verify normal heredocs have the right position", ->
source = '''
"""
a"""
'''
[stringToken] = CoffeeScript.tokens source
eq stringToken[2].first_line, 0
eq stringToken[2].first_column, 0
eq stringToken[2].last_line, 1
eq stringToken[2].last_column, 3

test "Verify heredocs ending with a newline have the right position", ->
source = '''
"""
a
"""
'''
[stringToken] = CoffeeScript.tokens source
eq stringToken[2].first_line, 0
eq stringToken[2].first_column, 0
eq stringToken[2].last_line, 2
eq stringToken[2].last_column, 2

test "Verify all tokens get a location", ->
doesNotThrow ->
tokens = CoffeeScript.tokens testScript
Expand Down

0 comments on commit b60b9ed

Please sign in to comment.