Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MsgExport's "Export All History" is very slow #609

Closed
LittleVulpix opened this issue Oct 25, 2016 · 0 comments
Closed

MsgExport's "Export All History" is very slow #609

LittleVulpix opened this issue Oct 25, 2016 · 0 comments
Assignees
@Robyer
Labels
Message export minor

Comments

@LittleVulpix
Copy link
Contributor

LittleVulpix commented Oct 25, 2016 •
edited
Loading

MsgExport has a functionality to export all history into text files.

However, this export is approximately 7-10x slower than for example history++'s message export.

In comparison:
History++ for my biggest contact: 26.6s
MsgExport for the same contact: 3 minutes 33seconds

That's eight times slower.

I understand there will always be some discrepancy because history++ only exports per contact whereas MsgExport is able to cleverly combine multiple contacts with the same nickname into one output file, thus grabbing a bit more data; but the contact in question has 90% of text in one protocol, and even so the time it takes to export all data seems too high.

I noticed that in processexplorer, you can see that msgexport constantly opens and closes the file, whereas history++ just flushes a bunch of data into a file and then closes it. Maybe implementing this type of "do it all in RAM and then flush it to a file" mode for the "Export All" function would be interesting.
@Robyer Robyer self-assigned this Dec 3, 2016
@LittleVulpix LittleVulpix mentioned this issue Jan 21, 2017
Closed
@Robyer Robyer closed this as completed in d8142b3 Feb 5, 2017
Robyer added a commit that referenced this issue Feb 5, 2017
@Robyer
Msg_Export: Optimize checking ignored contacts/protocols (addresses #609
5a3ac56 
)

Makes export about 20% faster.
Robyer added a commit that referenced this issue Feb 5, 2017
@Robyer
Msg_Export: Fix exporting unicode messages, make it even faster (addr…
c87d46c 
…esses #609)

Previously code worked with opening file and loading BOM at writing each event. When we stopped closing/opening file at each event it couldn't read BOM anymore, so it wrote messages in wrong codepage.

This is not fixed by parameter defining we are just appending the file, so it doesn't need to determing BOM or even check file size. Thanks to this exporting is now about 70% faster than the initial optimization commit (in sum 13.5x faster than before).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Robyer Robyer

Labels
Message export minor
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Robyer @aunsane @LittleVulpix