New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Test sprint] Big files editor freeze, ram eating, wrong config after app kill. #1025

Closed
fodrice opened this Issue Sep 20, 2018 · 17 comments

Comments

Projects
None yet
2 participants
@fodrice
Contributor

fodrice commented Sep 20, 2018

Long story short:

Make note. Copy paste big file (E. G 200 page file from libreoffice) then paste again, and again, go on 10-20 times. Ram consumption will rise to some big value (3 gb for me) and than everything freezes. It will unfreeze later but memory consumption remains at some big value.
After app shutdown it's freeze when I trying open created file (memory consumption is about 450m)
Also somehow after next shutdown after freeze I can't launch it any more /usr/bin/QOwnNotes just do nothing without any error message. Removing conf file from /home allowed to launch it again.

So there is a few bugs:

  1. There is a memory leak when copy text from clipboard.

  2. The whole notes file loads to memory that causes troubles with big files and troubles with the editor. It freezes when processing the really big file (tested on 50 mb markdown file).

  3. It seems the app didn't unload notes from memory (e g when switched to another note). Previous should be saved and freed from memory after autosave period. Testing with 7 mb markdown file shows that data loaded but not unloaded when switched to another notes file. Memory consumption just raised from 25mb to 100mb.
    You could use this file for memory check
    Elephants killing markdown editors.zip

  4. Nothing happens on crash with somehow made .conf file
    QOwnNotes.conf.zip

Possible fix for 2 and 3:

  • Separately load text files into a fixed-size chunks (e.g 10 000 symbols for one chunk).
  • Dynamically load/unload chunks to editor when needed. (It still could be stored in memory, to speed up )
  • Change editor behaviour to calculate the viewport and pre-read next and previous chunks.
  • Store undo/redo information on disk.
  • And finally add file size restriction.

Fix for 4:

Store last file/session info and settings separately. To avoid config corruption.

system info

QOwnNotes Version 18.09.2
Qt Version 5.9.6
Desktop environment: KDE
gentoo-linux 4.14.67

@pbek

This comment has been minimized.

Owner

pbek commented Sep 20, 2018

Thank you for our detailed report! But most of the things like the editor, undo/redo and clipboard are handled by the Qt framework directly and you would need to file bugs there...

@pbek

This comment has been minimized.

Owner

pbek commented Sep 20, 2018

Can you make out what part of the config made your app not starting again?

@fodrice

This comment has been minimized.

Contributor

fodrice commented Sep 20, 2018

This one
[General]
allowOnlyOneAppInstance=true

@pbek

This comment has been minimized.

Owner

pbek commented Sep 20, 2018

Thank you, seems to be a problem of the external library https://github.com/pbek/QOwnNotes/tree/develop/src/libraries/singleapplication.

@fodrice

This comment has been minimized.

@fodrice

This comment has been minimized.

Contributor

fodrice commented Sep 21, 2018

Checked memory again, every switch between big notes eats small amount of ram.

@pbek

This comment has been minimized.

Owner

pbek commented Sep 21, 2018

I am all open for Pull Request

@fodrice

This comment has been minimized.

Contributor

fodrice commented Sep 21, 2018

Added big markdown file I used for testing.
Elephants killing markdown editors.zip

pbek added a commit that referenced this issue Sep 22, 2018

@pbek

This comment has been minimized.

Owner

pbek commented Sep 22, 2018

18.09.3

  • attempted to improve the single application mode
    • on certain occasions it was possible that the app wouldn't start a 2nd
      time even if it was closed, this should be fixed now
@pbek

This comment has been minimized.

Owner

pbek commented Sep 22, 2018

As for testing with 32MB markdown files... I wonder that you have even manage to open such a big text file in QOwnNotes... It took quite a while even on a SSD.

pbek added a commit that referenced this issue Sep 22, 2018

@pbek

This comment has been minimized.

Owner

pbek commented Sep 22, 2018

There now is a new release, could you please test it and report if it works for you?

@pbek pbek added this to the 18.09.3 milestone Sep 22, 2018

@fodrice

This comment has been minimized.

Contributor

fodrice commented Sep 22, 2018

There now is a new release, could you please test it and report if it works for you?

I will test it tomorrow.

As for testing with 32MB markdown files

As I think the only way to improve this it to completely rewrite editor with MVC pattern. Render every line separately as view. And don't load the whole file but only beginning, load new file part when users need it (scrolls near it beginning or searching ... etc) and unloading unused parts. There will be lots of work... undo-redo, full-text search in file... and some more. Not Expected to be fixed soon.
But what really matters Is that after closing file memory remains in use.
Don't know c++ well. I dived into code a little and got a question. You load the whole text into database when file opened, don't you? Is it somehow unloaded when file is closed ?

@fodrice

This comment has been minimized.

Contributor

fodrice commented Sep 22, 2018

Single application mode now works fine for me. Tested latest appimage.

I wonder that you have even manage to open such a big text file in QOwnNotes

It's only 35mb it should be loaded to memory in less than 1 second...
It depends more on Processor, system bus, and memory speed than HDD speed.
And, obliviously, it depends on application. Cause app have to render that stuff. It opens in about 10 seconds on my system. (i5-2nd_gen,4cores,3.10GHz; Nvidia-970; gentoo-linux-4.14.67 kernel).
Meanwhile Kate (kde editor) opens the same file instantly on the same system.
But as I wrote, just add it to your "big things todo later" list.

@pbek

This comment has been minimized.

Owner

pbek commented Sep 23, 2018

You load the whole text into database when file opened, don't you? Is it somehow unloaded when file is closed ?

all note files are loaded into an in-memory sqlite database in the beginning. if the note is opened the note's text is loaded in a QPlainTextEdit

It's only 35mb it should be loaded to memory in less than 1 second...

I don't think that isn't the issue, I think the issue is editing so much text in a QPlainTextEdit

@pbek

This comment has been minimized.

Owner

pbek commented Sep 26, 2018

@fodrice did 18.09.3 bring any improvement with application locking after a crash?

@fodrice

This comment has been minimized.

Contributor

fodrice commented Sep 26, 2018

Yes. Now it works fine.

@pbek

This comment has been minimized.

Owner

pbek commented Sep 26, 2018

Thank you for testing! Since I cannot do much about the other stuff I will close this for now...

@pbek pbek closed this Sep 26, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment