Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
Speed up CSV import by not querying the stream position
Avoid querying the position in the text stream using Qt's pos() function
to update the progress dialog. Instead keep track of the stream position
manually. This is possible here because we don't ever seek in the file.
In result, this speeds up the CSV import dramatically.
  • Loading branch information
MKleusberg committed Nov 5, 2017
1 parent a03a901 commit ed9fda2
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion src/csvparser.cpp
Expand Up @@ -140,9 +140,11 @@ CSVParser::ParserResult CSVParser::parse(csvRowFunction insertFunction, QTextStr
};
FieldBufferDealloc dealloc(record);

qint64 bufferPos = 0;
while(!stream.atEnd())
{
sBuffer = stream.read(m_nBufferSize).toUtf8();
bufferPos += sBuffer.length();
auto sBufferEnd = sBuffer.constEnd();

for(auto it = sBuffer.constBegin(); it != sBufferEnd; ++it)
Expand Down Expand Up @@ -276,7 +278,7 @@ CSVParser::ParserResult CSVParser::parse(csvRowFunction insertFunction, QTextStr

if(m_pCSVProgress && parsedRows % 100 == 0)
{
if(!m_pCSVProgress->update(stream.pos()))
if(!m_pCSVProgress->update(bufferPos))
return ParserResult::ParserResultCancelled;
}
}
Expand Down

1 comment on commit ed9fda2

@MKleusberg
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With this we're now more or less as fast as the sqlite3 CLI tool import :smile: I haven't done any thorough testing yet but: at least for the postcode file, and at least on my system, and at least for the one try I gave it, we were now about 5s faster than the CLI tool (while offering more config options and a progress bar) :wink:

Please sign in to comment.