Incorrect line numbering/location information when certain unicode characters are used

[mivok]

This is a strange one:

$ cat testcase.rb
#!/usr/bin/env ruby
require 'kramdown'
text = <<EOF
nbsp nbsp (the first space on this line is unicode 0xA0 - a non-breaking space)

- Test

# Test
EOF

doc = Kramdown::Document.new(text).root
puts doc.children.inspect

$ ./testcase.rb
[<kd:p nil {:location=>1} [<kd:text "nbsp nbsp" nil>]>, <kd:blank "\n" nil>, <kd:ul nil {:location=>3} [<kd:li nil {:location=>3} [<kd:p nil {:location=>3, :transparent=>true} [<kd:text "Test" nil>]>]>]>, <kd:blank "\n" nil>, <kd:header nil {:level=>1, :raw_text=>"Test", :location=>6} [<kd:text "Test" nil>]>]

The location for the header is parsed as line 6, when it should be line 5. Interestingly, the location for the list item is shown as line 3, which is correct.

• If I delete the 0xA0 (http://www.fileformat.info/info/unicode/char/a0/index.htm) character in between the two 'nbsp' words on the first line, and replace with a regular space it works.
• If I delete the list item, it works.

This came from a much larger markdown document, and I managed to get it down to that small test case while still reproducing the issue.

[mivok]

I've managed to reproduce with other unicode characters as well, so it's not just the non-breaking space:

#!/usr/bin/env ruby
require 'kramdown'
text = <<EOF
®

- Test

# Test
EOF

doc = Kramdown::Document.new(text).root
puts doc.children.inspect

[mivok]

I just spotted https://github.com/gettalong/kramdown/blob/master/lib/kramdown/utils/string_scanner.rb#L33 which mentions that this could be an issue with older rubies so I wanted to point out that this is in ruby 2.1.1:

$ ruby -e 'require "strscan";p StringScanner.new("foo", 1).respond_to?(:charpos)'
true

[mivok]

Interestingly, changing best_pos to pos in https://github.com/gettalong/kramdown/blob/master/lib/kramdown/utils/string_scanner.rb#L64 causes the line numbers to display correctly, and I ran the testcase in ruby 1.9.3 (after adding # encoding: utf-8 to the script) and it also showed the correct line number. Unfortunately I'm not sure what the implications of this are (presumably there is a good reason for using charpos/best_pos), but it should point you in the right direction.

[gettalong]

Thanks for the detailed bug report! You are completely right!

The StringScanner#charpos method returns the character position whereas the StringScanner#pos method returns the byte position. When setting the position with StringScanner#pos= (see http://www.ruby-doc.org/stdlib-2.0.0/libdoc/strscan/rdoc/StringScanner.html#method-i-pointer-3D) the byte position is needed.

So the #best_pos method is not needed, #pos is sufficient.

[jhund]

@gettalong I'm happy to submit a pull request to fix this issue since we rely on it in Repositext. Should I go ahead?

[gettalong]

@jhund Thanks but I have already implemented the changes, I will just need to push a new release which will be done in the next few days.

[jhund]

@gettalong great, thanks for fixing the location bug.