Massive CPU use in LibreOffice with a long ODT document

[mestaritonttu]

I was wondering, if you could determine, if this problem is due to LT or LibO: https://bugs.documentfoundation.org/show_bug.cgi?id=115757
There is a callgrind trace attached.

[aviwad]

Hello! I encountered this issue with a 17-page word doc! I cannot use LanguageTool if this issue continues 😢

[danielnaber]

Is this reproducible, i.e. does it happen every time you try? Can you provide the document?

[aviwad]

It happens every time I try, and I leave it for 10 minutes. Thing is, it is my Personal Project document which cannot be shared right now. But it was 17 pages long, with tons of images and text. I will try making a similar generic document which can be shared.

[danielnaber]

I will try making a similar generic document which can be shared.

Yes, that would be great. Please also let us know which version of LT and LibreOffice you're using.

[aviwad]

Here is a dummy document with all text replaced with generic stuff.
LanguageTool version: 4.3 LibreOffice extension
LibreOffice version: 6.1.2.1 Fedora 29 (Not flatpak)
(the attached file is equivalent to my project, but stuffed with lorem ipsum, and cat pictures)
https://transfer.sh/zqh4D/dummy%20document%20for%20github.docx

[danielnaber]

Is there any specific editing you need to do to trigger the issue? I'm asking because I cannot reproduce with LO Version: 6.0.2.1 (under Ubuntu) and LT 4.3.

[pseudo-morph]

I too can confirm extraordinarily high CPU usage when using LanguageTool (4.5.1) with LibreOffice (6.2.2) on linux (Ubuntu 18.10 / 19.04). See the two bugs linked below for more information.

https://bugs.documentfoundation.org/show_bug.cgi?id=124346
https://bugs.launchpad.net/ubuntu/+source/libreoffice/+bug/1825513

For me, the issue arises when I open a large document in LibreOffice. This causes the CPU to spike to somewhere between 100-200% for a sustained period. For example, when loading my PhD thesis with LanguageTool installed CPU usage remained at above 100% for around 7 minutes, before settling to regular levels. Without LanguageTool installed the CPU spike when loading the document was <30 seconds.

I didn't experience the issue with earlier versions of LibreOffice and LanguageTool.

This issue is reproducible always when LangueageTool is installed. I can't download Avi's file to test, but see the linked bug for a document of lorem ipsum I've been using to test.

[danielnaber]

With the larger one of your Lorem.odt files, I get some load:

@FredKruse Any idea if getNumberOfAllTextParagraphs() could be optimized? It seems to take about 30% of all CPU time.

[FredKruse]

I can confirm this issue. It happens if a big document is analyzed after loading the document. The interface of LO gives the document paragraph by paragraph (and within sentence by sentence) to LT. The office extension of LT is restricted to one thread. So the usage of CPU is restricted to one core, but this is want the application needs. We could only try to shorten the time it takes.
I'm a bit busy, but I will do some tests the next days.
@danielnaber: Is there a possibility to limit the threads in the standalone version to one? I got the idea to compare the time the standalone version needs to that of the LO version. I fear that LT needs this time to analyze a big document.

[danielnaber]

@FredKruse In your IDE, you could replace new MultiThreadedJLanguageTool by new JLanguageTool in e.g. commandline/Main.java and you should get one thread.

[FredKruse]

I did some tests. I did it on my not so new 2-core notebook. As long as LT analysis runs, one core is fully occupied by LT. I compared the results with the command line application (reduced of one thread). The used text needs 185 seconds by the command line application.
When I did an analysis of the text within LO and use the "only one paragraph" option it needs 16% more time for the analysis (This is the absolute minimum given by the LT interface). There is an additional time of nearly 1 minute when the text is loaded at first, I don't know what LO does in this period of time. The default LT option of analysis of 5 paragraphs increases the time by 50% compared with the command line application. If you use the full text option it increases to 43%.
A change of the number of paragraphs changes not much, because of the internal process of do the first iteration of analysis. I could try to do some optimization, but it will always be more than 43%. And there are some more factors then the length of the period for the first check (e.g. how long it needs to show the first mark). So I'm not very optimistic.
I think, we have to do a decision where we set priorities. A good analysis of the text costs CPU time. Form my point of view we should accept this and don't decrease the quality of LT check.

[Black56651]

Da ich den Stein ins Rollen gebracht habe, wird es glaube ich mal Zeit, dass ich dazu was sage.
Ich habe mehrere LO und LT Sprünge hinter mir und mir ist dabei immer aufgefallen, dass LT mit höherern LO-Versionen immer neue Probleme dazukamen, zumindest bei mir. Plötzlich gab es Bugs, die es mit älteren Versionen es nicht gab. Wodurch sie ausgelöst waren, weiß ich nicht. Doch irgendwann wurde es so unstabil, dass ich mir einem Rat im LT-Forum gesucht habe.

Da meine Versuche beim Bugzilla-Forum vom LO gescheichtert sind, weil ich kein Englisch spreche, zu schweige schreiben kann. Ich habe zunächst Freunde gefragt, die mir immer beim Übersetzen geholfen haben. Da aber sich nichts geändert hat, habe ich auf gegeben. Der Link zum Bugzilla-Beitrag wurde von mir verfasst.

Ich habe diesem Rat auch in der Tat umgesetzt, aber wenn sogar kleine bis mittlere große Dokumente starke bis brachiale Performanzprobleme haben, frage ich mich, ob LO in Sachen Performanz immer schlechteroder ob mein PC immer weniger supportet wird.

Wenn ich helfen kann, lass es mich wissen. Ihr wisst ja, dass ich dazu bereit bin, zu helfen, solange ich es kann.

Grüße Dallun511

[FredKruse]

There was done a lot of optimization and bug-fixing since opening the issue. The performance problems are fixed as far as I see. The performance problem with getNumberOfAllTextParagraphs() is solved too.
@mestaritonttu: Please close the issue.