BOM UTF-8 is ignored after rewind

[diei]

I have a CSV file with "forced quotes" and UTF-8 BOM (\xEF\xBB\xBF) which CSV can not read after a rewind. I get "CSV::MalformedCSVError: Illegal quoting in line 1."

My UTF-8 CSV file with BOM:

File.open('bom_test.csv', 'w') do |io|
  io.write("\xEF\xBB\xBF\"Name\",\"City\"\n\"John Doe\",\"New York\"")
end

Reproduce error:

# Case 1
csv = CSV.open('bom_test.csv', 'r:BOM|UTF-8', {headers: true})
csv.shift
# => #<CSV::Row "Name":"John Doe" "City":"New York">
csv.rewind
csv.shift
# => CSV::MalformedCSVError (Illegal quoting in line 1.)

# Case 2
csv = CSV.open('bom_test.csv', 'r:BOM|UTF-8', {headers: true})
csv.readline
# => #<CSV::Row "Name":"John Doe" "City":"New York">
csv.rewind
csv.readline
# => CSV::MalformedCSVError (Illegal quoting in line 1.)

[kou]

This library rely on Ruby's BOM processing.
It seems that Ruby's BOM processing doesn't support rewind:

File.open("/tmp/a.txt", "w") do |x|
  x.puts("\xEF\xBB\xBFa,b,c")
end
File.open("/tmp/a.txt", "r:BOM|UTF-8") do |x|
  p x.gets.unpack("U*") # => [97, 44, 98, 44, 99, 10]
  x.rewind
  p x.gets.unpack("U*") # => [65279, 97, 44, 98, 44, 99, 10]
end

Could you report this to Ruby's issue tracking system? https://bugs.ruby-lang.org/

[diei]

I have opened the issue https://bugs.ruby-lang.org/issues/16143 for this bug.

[kou]

Thanks.
This is an upstream problem. So I close this.