Permalink
Browse files

Speed up CSV import by not querying the stream position

Avoid querying the position in the text stream using Qt's pos() function
to update the progress dialog. Instead keep track of the stream position
manually. This is possible here because we don't ever seek in the file.
In result, this speeds up the CSV import dramatically.
  • Loading branch information...
MKleusberg committed Nov 5, 2017
1 parent a03a901 commit ed9fda28ea6b54bf86bf78ceaaae3256f3dec3e5
Showing with 3 additions and 1 deletion.
  1. +3 −1 src/csvparser.cpp
@@ -140,9 +140,11 @@ CSVParser::ParserResult CSVParser::parse(csvRowFunction insertFunction, QTextStr
};
FieldBufferDealloc dealloc(record);

qint64 bufferPos = 0;
while(!stream.atEnd())
{
sBuffer = stream.read(m_nBufferSize).toUtf8();
bufferPos += sBuffer.length();
auto sBufferEnd = sBuffer.constEnd();

for(auto it = sBuffer.constBegin(); it != sBufferEnd; ++it)
@@ -276,7 +278,7 @@ CSVParser::ParserResult CSVParser::parse(csvRowFunction insertFunction, QTextStr

if(m_pCSVProgress && parsedRows % 100 == 0)
{
if(!m_pCSVProgress->update(stream.pos()))
if(!m_pCSVProgress->update(bufferPos))
return ParserResult::ParserResultCancelled;
}
}

1 comment on commit ed9fda2

@MKleusberg

This comment has been minimized.

Copy link
Member Author

MKleusberg commented on ed9fda2 Nov 6, 2017

With this we're now more or less as fast as the sqlite3 CLI tool import 😄 I haven't done any thorough testing yet but: at least for the postcode file, and at least on my system, and at least for the one try I gave it, we were now about 5s faster than the CLI tool (while offering more config options and a progress bar) 😉

Please sign in to comment.