Skip to content

Commit

Permalink
[ruby/csv] parser: use row separator explicitly
Browse files Browse the repository at this point in the history
It will improve performance a bit. (But I haven't confirmed it yet...)

ruby/csv@06a65b0302
  • Loading branch information
kou committed Dec 24, 2021
1 parent 4a5d372 commit 002ce9f
Showing 1 changed file with 5 additions and 3 deletions.
8 changes: 5 additions & 3 deletions lib/csv/parser.rb
Expand Up @@ -85,9 +85,10 @@ def keep_drop
# If there is no more data (eos? = true), it returns "".
#
class InputsScanner
def initialize(inputs, encoding, chunk_size: 8192)
def initialize(inputs, encoding, row_separator, chunk_size: 8192)
@inputs = inputs.dup
@encoding = encoding
@row_separator = row_separator
@chunk_size = chunk_size
@last_scanner = @inputs.empty?
@keeps = []
Expand Down Expand Up @@ -250,7 +251,7 @@ def read_chunk
@last_scanner = @inputs.empty?
true
else
chunk = input.gets(nil, @chunk_size)
chunk = input.gets(@row_separator, @chunk_size)
if chunk
raise InvalidEncoding unless chunk.valid_encoding?
@scanner = StringScanner.new(chunk)
Expand Down Expand Up @@ -778,6 +779,7 @@ def build_scanner
Integer((ENV["CSV_PARSER_SCANNER_TEST_CHUNK_SIZE"] || "1"), 10)
InputsScanner.new(inputs,
@encoding,
@row_separator,
chunk_size: chunk_size)
end
else
Expand Down Expand Up @@ -807,7 +809,7 @@ def build_scanner
StringIO.new(sample)
end
inputs << @input
InputsScanner.new(inputs, @encoding)
InputsScanner.new(inputs, @encoding, @row_separator)
end
end
end
Expand Down

0 comments on commit 002ce9f

Please sign in to comment.