Parsing issue reading from a stream >64KB on Windows

[RedBreadcat]

Hello,
I'm encountering a parsing issue on Windows files >64KB. This has occurred across several different machines. GetRowCount (and rapidcsv::Document's internal mData member) have more rows than is present in the file.
I have seen this issue on the latest master ee33419, as well as release 8.53.

I believe that this is related to the way that the MSVC standard library implementation magically removes \r characters when reading a file.
In void ReadCsv(std::istream& pStream), the file size calculated from std::streamsize length = pStream.tellg(); includes \r characters as one would expect.

However, in void ParseCsv(std::istream& pStream, std::streamsize p_FileLength), calls to pStream.read(buffer.data(), readLength); return a buffer that does not contain any \r characters. So in short, the total amount of bytes outputted by read is smaller than the size of the file.
Because of this mismatch, parts of buffer are parsed more than is required, leading to multiple additional rows being appended to mData.

Below is some self-contained code that should reproduce the issue:

#include <fstream>
#include <iostream>

#include "rapidcsv.h"

int dataRows = 5000;
const auto filename = "testfile.csv";

void CreateFile()
{
   std::ofstream out(filename);
   out << "Foo,Bar,Baz" << std::endl;
   for (int i = 0; i < dataRows; i++)
   {
      out << i << "," << i * 2 << "," << i * 3 << std::endl;
   }
}

void ReadFile()
{
   std::ifstream file(filename);
   
   rapidcsv::Document doc(file);
   std::cout << doc.GetRowCount() << std::endl; // ==5353 in my case
}

int main()
{
   CreateFile();
   ReadFile();

   return 0;
}

[d99kris]

Hello,
Thanks for reporting the issue and providing the detailed analysis! It's a pretty bad bug and I'm surprised it's not been encountered (or detected) before.

I can confirm your test case reproduces the issue on MSVC for me as well. I'm preparing a fix in rapidcsv (incl. updates to documentation and test cases) and will likely get it in this weekend.

Meanwhile, if you need to work around it, I think there are two options:

1. Open the file using rapidcsv, i.e. change ReadFile() to:

void ReadFile()
{
   rapidcsv::Document doc(filename);
   std::cout << doc.GetRowCount() << std::endl;
}

2. Open the ifstream in binary mode, i.e. change ReadFile() to:

void ReadFile()
{
   std::ifstream file(filename, std::ios::binary);
   rapidcsv::Document doc(file);
   std::cout << doc.GetRowCount() << std::endl;
}

[RedBreadcat]

Great, thank you for the ergonomic suggestions to work around the issue in the meantime!

[d99kris]

Hello,
The reported issue has been addressed in above commit. Please let me know if you encounter any issues. Thanks!

[ezemskov]

Thanks Kris,
rapidcsv 8.61 fixes our issue.