Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Slow startup when using language_tool_python in the test python setup #862

Closed
belug23 opened this issue Apr 11, 2021 · 6 comments
Closed

Slow startup when using language_tool_python in the test python setup #862

belug23 opened this issue Apr 11, 2021 · 6 comments
Labels
bug
Milestone
0.12.0

Comments

@belug23
Copy link
Contributor

belug23 commented Apr 11, 2021

I'm using Manuskript in my development python setup and I have a strange bug with the LanguageTool library. It's taking a really long time to start and I have a strange log.

The environment is setup like described in package/prepare_linux.sh with the extra pip install language_tool_python to have a spellchecker.

For the moment I have no idea how to fix this, so any help is welcome.

Here's my startup log:

Logging to STDERR.
Logging to file: /home/belug/.local/share/manuskript/manuskript/manuskript.log
Operating System: Linux-5.8.0-7642-generic-x86_64-with-debian-bullseye-sid
Hardware: x86_64 / x86_64
Running from unpackaged source code.
* sys.executable = '/home/belug/.pyenv/versions/3.6.7/bin/python'
* sys.argv = ['/home/belug/dev/manuskript/bin/manuskript']
* sys.path = ['/home/belug/dev/manuskript/bin',
 '/home/belug/dev/manuskript/bin/..',
 '/home/belug/.pyenv/versions/3.6.7/lib/python36.zip',
 '/home/belug/.pyenv/versions/3.6.7/lib/python3.6',
 '/home/belug/.pyenv/versions/3.6.7/lib/python3.6/lib-dynload',
 '/home/belug/.pyenv/versions/3.6.7/lib/python3.6/site-packages']
* sys.prefix = '/home/belug/.pyenv/versions/3.6.7'
Manuskript 0.11.0#b72098d (Python 3.6.7)
* PyQt 5.9 (compiled against Qt 5.9.1)
  * Qt 5.9.1 (runtime)
* lxml.etree 4.6.3.0
  * libxml   2.9.10 (compiled: 2.9.10)
  * libxslt  1.1.34 (compiled: 1.1.34)
* pyEnchant N/A (libenchant: N/A)
* pySpellChecker N/A
* Symspellpy N/A
* Markdown N/A
Web rendering engine: QtWebEngine
Preferred translation: builtin (based on user setting)
Using the builtin translation. (U.S. English)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)

openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
Loading: /home/belug/writings/test.msk
Detected file format version: 1. Zip: True.
Loading /home/belug/writings/test.msk (zip)
Reading labels:
* Add status: Idea (#ffff00)
* Add status: Note (#00ff00)
* Add status: Chapter (#0000ff)
* Add status: Scene (#ff0000)
* Add status: Research (#00ffff)
Reading status:
* Add status: TODO
* Add status: First draft
* Add status: Second draft
* Add status: Final
Reading plots:
Reading World:
Reading Characters:
Reading outline:
* Adds 00-Something_1.md to Root (file)
* Adds 01-Something_2.md to Root (file)
* Adds 02-Something_3.md to Root (file)
* Adds 03-Something_4.md to Root (file)
* Adds 04-Something_5.md to Root (file)
* Adds 05-Something_6.md to Root (file)
* Adds 06-Something_7.md to Root (file)
* Adds 07-Something_8.md to Root (file)
* Adds 08-Something_9.md to Root (file)
* Adds 09-Something_10.md to Root (file)
Project /home/belug/writings/test.msk loaded.
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)

Running it in debug with VSCode I see that I have a lot of processes:
image

Also when checking netstat that I have a lot of languageTool servers running:

tcp6       0      0 127.0.0.1:8090          :::*                    LISTEN      1000       12666468   851541/java         
tcp6       0      0 127.0.0.1:8091          :::*                    LISTEN      1000       12678620   852398/java         
tcp6       0      0 127.0.0.1:8092          :::*                    LISTEN      1000       12681936   853377/java         
tcp6       0      0 127.0.0.1:8093          :::*                    LISTEN      1000       12678801   854194/java         
tcp6       0      0 127.0.0.1:8094          :::*                    LISTEN      1000       12685845   855004/java         
tcp6       0      0 127.0.0.1:8095          :::*                    LISTEN      1000       12684880   855882/java         
tcp6       0      0 127.0.0.1:8096          :::*                    LISTEN      1000       12695785   856811/java         
tcp6       0      0 127.0.0.1:8097          :::*                    LISTEN      1000       12695938   857806/java         
tcp6       0      0 127.0.0.1:8098          :::*                    LISTEN      1000       12694299   858847/java         
tcp6       0      0 127.0.0.1:8099          :::*                    LISTEN      1000       12711130   859930/java         
tcp6       0      0 127.0.0.1:8100          :::*                    LISTEN      1000       12704231   861080/java         
tcp6       0      0 127.0.0.1:8101          :::*                    LISTEN      1000       12704675   862295/java         
tcp6       0      0 127.0.0.1:8102          :::*                    LISTEN      1000       12725452   863565/java         
tcp6       0      0 127.0.0.1:8103          :::*                    LISTEN      1000       12731512   864888/java         
tcp6       0      0 127.0.0.1:8104          :::*                    LISTEN      1000       12717993   866266/java         
tcp6       0      0 127.0.0.1:8105          :::*                    LISTEN      1000       12738884   867707/java         
tcp6       0      0 127.0.0.1:8106          :::*                    LISTEN      1000       12736391   869199/java         
tcp6       0      0 127.0.0.1:8107          :::*                    LISTEN      1000       12752424   870743/java         
tcp6       0      0 127.0.0.1:8108          :::*                    LISTEN      1000       12755894   872346/java         
tcp6       0      0 127.0.0.1:8109          :::*                    LISTEN      1000       12761758   874036/java         
tcp6       0      0 127.0.0.1:8110          :::*                    LISTEN      1000       12775929   875711/java         
tcp6       0      0 127.0.0.1:8111          :::*                    LISTEN      1000       12787036   877452/java         
tcp6       0      0 127.0.0.1:8112          :::*                    LISTEN      1000       12805215   879263/java         
tcp6       0      0 127.0.0.1:8113          :::*                    LISTEN      1000       12815502   881033/java         
tcp6       0      0 127.0.0.1:8081          :::*                    LISTEN      1000       8222272    490821/java         
tcp6       0      0 127.0.0.1:8114          :::*                    LISTEN      1000       12823718   882865/java         
tcp6       0      0 127.0.0.1:8082          :::*                    LISTEN      1000       12663983   847385/java         
tcp6       0      0 127.0.0.1:8115          :::*                    LISTEN      1000       12824975   884770/java         
tcp6       0      0 127.0.0.1:8083          :::*                    LISTEN      1000       12658199   847583/java         
tcp6       0      0 127.0.0.1:8116          :::*                    LISTEN      1000       12843728   886647/java         
tcp6       0      0 127.0.0.1:8084          :::*                    LISTEN      1000       12666921   847838/java         
tcp6       0      0 127.0.0.1:8085          :::*                    LISTEN      1000       12666000   848121/java         
tcp6       0      0 127.0.0.1:8086          :::*                    LISTEN      1000       12643892   848546/java         
tcp6       0      0 127.0.0.1:8087          :::*                    LISTEN      1000       12658597   849118/java
@TheJackiMonster
Copy link
Collaborator

That is weird... On my system it only starts two Java processes with the LanguageTool server. I just thought one of it is a backup in case of a crash or to reduce latency but starting 34 processes shouldn't happen.

The problem is that this behavior is part of the Python API for LanguageTool. So this issue should probably be forwarded to the API: https://github.com/jxmorris12/language_tool_python

Otherwise it could be a problem in Python itself. Calling the helper function get_languagetool_languages() has to create a new LanguageTool object to query the available languages. So every call a new instance with an own server and a separate process will be created. I will try to fix this point of failure with a global LanguageTool instance which will be used instead of a new one. Let's hope this fixes most of the issue on our application side. ^^'

@belug23
Copy link
Contributor Author

belug23 commented Apr 11, 2021

If I start it from the system install (System python 3.8.8) I only have 2 servers too, but if I use the test environment (Python 3.6.7) setup like in the package/prepare_linux.sh file, I get 34 processes.... really strange, so I believe it's more linked to manuskript at the moment.
There's a part where language_tool_python should be able to support threading, but it's strange that in some conditions Manuskript starts that many threads.

Also, with a breakpoint all the checkText are called with empty strings.

@belug23
Copy link
Contributor Author

belug23 commented Apr 11, 2021

Thanks for the hint about get_languagetool_languages() That's totally what is causing the problem.

@TheJackiMonster are you fixing it now? Because I can do it now that I know what's causing it.

@TheJackiMonster
Copy link
Collaborator

TheJackiMonster commented Apr 11, 2021 •
edited

It should be fixed now. It only starts one Java process and changes language of one instance.

Edit: Please check if it's still causing issues before closing the issue. ^^'

@TheJackiMonster TheJackiMonster added this to the 0.12.0 milestone Apr 11, 2021
@belug23
Copy link
Contributor Author

belug23 commented Apr 11, 2021

I still have a lot of logs for:

openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)

But now it's fast and I can see that the server is started only once, thanks for the fix.

@belug23 belug23 closed this as completed Apr 11, 2021
@TheJackiMonster
Copy link
Collaborator

I still have a lot of logs for:

openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)

Yes, I also have a lot of logged messages from OpenJDK. I'm still not sure if that's an issue of the Python API (maybe starting a lot of processes in short delays for example) or if that's an issue of OpenJDK (logging these lines for each thread in the process or similar). ^^'

Maybe this will change in the future with newer releases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
0.12.0
Development

No branches or pull requests
2 participants
@belug23 @TheJackiMonster