Bad quote rule detection for a backslash-escaped test file

[st-pasha]

This is the file:

"C\\D"
AB\x20CD\n
"\"one\", \'two\', three"
"\r\t\v\a\b\071\uABCD"

And this is how it behaves with fread:

> fread('"C\\\\D"\nAB\\x20CD\\n\n"\\"one\\", \\\'two\\\', three"\n"\\r\\t\\v\\a\\b\\071\\uABCD"\n', verbose=TRUE)
Input contains a \n or is "". Taking this to be text input (not a filename)
[01] Check arguments
  Using 8 threads (omp_get_max_threads()=8, nth=8)
  NAstrings = [<<NA>>]
  None of the NAstrings look like numbers.
  show progress = 1
  0/1 column will be read as integer
[02] Opening the file
  `input` argument is provided rather than a file name, interpreting as raw text to read
[03] Detect and skip BOM
[04] Arrange mmap to be \0 terminated
  \n has been found in the input and different lines can end with different line endings (e.g. mixed \n and \r\n in one file). This is common and ideal.
[05] Skipping initial rows if needed
  Positioned on line 1 starting: <<"C\\D">>
[06] Detect separator, quoting rule, and ncolumns
  Detecting sep automatically ...
  Detected 1 columns on line 1. This line is either column names or first data row. Line starts as: <<"C\\D">>
  Quote rule picked = 0
  fill=false and the most number of columns found is 1
[07] Detect column types, good nrow estimate and whether first row is column names
  Number of sampling jump points = 2 because (67 bytes from row 1 to eof) / (2 * 67 jump0size) == 0
  A line with too-many fields (1/1) was found on line 2 of sample jump 0. 
  Type codes (jump 000)    : A  Quote rule 0
  'header' determined to be true because all columns are type string and a better guess is not possible
  All rows were sampled since file is small so we know nrow=1 exactly
[08] Assign column names
[09] Apply user overrides on column types
  After 0 type and 0 drop user overrides : A
[10] Allocate memory for the datatable
  Allocating 1 column slots (1 - 0 dropped) with 1 rows
[11] Read the data
  jumps=[0..1), chunk_size=1048576, total_size=60
  Restarting team from jump 0. nSwept==0 quoteRule==1
  jumps=[0..1), chunk_size=1048576, total_size=60
  Too few rows allocated. Allocating additional 1024 rows (now nrows=1025) and continue reading from jump 0
  jumps=[0..1), chunk_size=1048576, total_size=60
Read 3 rows x 1 columns from 67 bytes file in 00:00.000 wall clock time
[12] Finalizing the datatable
  Type counts:
         1 : string    'A'
=============================
   0.000s ( 16%) Memory map 0.000GB file
   0.000s ( 41%) sep='' ncol=1 and header detection
   0.000s (  5%) Column type detection using 1 sample rows
   0.000s (  3%) Allocation of 1025 rows x 1 cols (0.000GB) of which 3 (  0%) rows used
   0.000s ( 36%) Reading 1 chunks (0 swept) of 1.000MB (-2147483648 rows) using 1 threads
   +    0.000s (  1%) Parse to row-major thread buffers (grown 0 times)
   +    0.000s (  2%) Transpose
   +    0.000s ( 33%) Waiting
   0.000s (  0%) Rereading 0 columns due to out-of-sample type exceptions
   0.000s        Total
                        C\\\\D
                        <char>
1:                AB\\x20CD\\n
2: \\"one\\", \\'two\\', three
3: \\r\\t\\v\\a\\b\\071\\uABCD
Warning message:
In fread("\"C\\\\D\"\nAB\\x20CD\\n\n\"\\\"one\\\", \\'two\\', three\"\n\"\\r\\t\\v\\a\\b\\071\\uABCD\"\n",  :
  Found and resolved improper quoting. First healed line 3: <<"\"one\", \'two\', three">>

in the end the file is read correctly (except that escaped symbols are not unescaped), but there are still notable irregularities:

• In section [06] we see Quote rule picked = 0, even though it should have been QR 1,
• In section [07]: A line with too-many fields (1/1) was found on line 2 of sample jump 0., caused by having wrong quote rule,
• In section [07]: All rows were sampled since file is small so we know nrow=1 exactly: in fact we should have had nrows=3
• In section [11]: unnecessary reallocation, twice; and re-reading of the input
• In section [12]: (-2147483648 rows) reported
• Finally, a warning is produced about "Found and resolved improper quoting". This warning should not have been there: the data is quoted correctly. This warning could have become an error, if configured so in R.