improve perfomance, using StringBuilder instead of string concate #7
Conversation
|
Yes, should you have a wide table with small values then StringBuilder can definitely have a huge impact. I'm actually surprised we haven't caught this before. Thanks for the catch! NB! be aware your PR was non-compiling due to missing using-statement and also changed the expectations about handling missing/empty values (3 tests broken). Fixed these issues in develop branch 8cb19791462171358212e58745080c496d75d0fca). FYI, current algorithm is still simplistic and targeting mostly csv files which are relatively narrow and streaming toolset is designed to help handling files with many rows. Regardless, parsing 100MB using streaming API (parser.ReadNextRow()) should not have taken 15 mins. I would appreciate if you could make a repro file available to me for further investigation on handling such structures.. |
|
can you provide me your email, because my csv file have some sensitive data, cannot public here |
|
Send the link here: imre.pyhvel@gmail.com |
|
I forgot that the problem is not large file but the large filed! |
|
Regardless, if possible do provide the example file. 15mins is unacceptable. I assume this should be some interesting edge case where single fields were split up into too many pieces during lexing phase, A typical use case would have no string concatenations in that step. |
|
I am surprised, too. I try with OpenCSV in Java and it takes more than 20 minutes. I will send you my file in next week! |
I have a 100MB csv file, it takes 15 minutes to read this file. After read the source code I see that using string concate is very terible, I replace it with StringBuilder. Now it takes only few seconds compare to 15 minutes to read this file!