Got wrong results when parse from File and that includes 3 bytes char with skip_lines

[ff2248]

I haven't figured out how to solve that problem, but i found some step to reproduce it:

• read data from File.open
• skip_lines options are presence
• data length is over the chunk size (32 * 1024)
• data includes 3 bytes char (such as chinese)
• v3.0.1 works correctly, It's happening in v3.0.2 and afters

test case:

def test_three_bytes_chars
  Tempfile.create(['temp', '.csv']) do |tempfile|
    tempfile.close
    path = tempfile.path

    text = "\xE5\x93\x88\xE5\x9B\x89"
    File.open(path, "w") do |csv|
      row = [text].join(',').concat("\n")
      row_count = (32 * 1024) / (StringIO.new(row).length) + 1
      csv << (row * row_count)
    end;
    assert_equal(
      [text],
      CSV.read(File.open(path), headers: true, :skip_lines => /\A#/).to_a.last
    )
  end
end

Results in Ruby 2.5.5:

# Running tests:

[102/756] TestCSVFeatures#test_three_bytes_chars = 0.06 s
  1) Failure:
TestCSVFeatures#test_three_bytes_chars [/Users/jeff/project/csv/test/csv/test_features.rb:341]:
<["哈囉"]> expected but was
<["囉"]>.

[142/756] TestCSVFeatures::DifferentOFS#test_three_bytes_chars = 0.06 s
  2) Failure:
TestCSVFeatures::DifferentOFS#test_three_bytes_chars [/Users/jeff/project/csv/test/csv/test_features.rb:341]:
<["哈囉"]> expected but was
<["囉"]>.

Finished tests in 2.863357s, 264.0258 tests/s, 2669.9430 assertions/s.
756 tests, 7645 assertions, 2 failures, 0 errors, 0 skips

ruby -v: ruby 2.5.5p157 (2019-03-15 revision 67260) [x86_64-darwin18]

[kou]

Thanks for your report.
I've fixed.