[Test sprint] Big files editor freeze, ram eating, wrong config after app kill.

[fodrice]

Long story short:

Make note. Copy paste big file (E. G 200 page file from libreoffice) then paste again, and again, go on 10-20 times. Ram consumption will rise to some big value (3 gb for me) and than everything freezes. It will unfreeze later but memory consumption remains at some big value.
After app shutdown it's freeze when I trying open created file (memory consumption is about 450m)
Also somehow after next shutdown after freeze I can't launch it any more /usr/bin/QOwnNotes just do nothing without any error message. Removing conf file from /home allowed to launch it again.

So there is a few bugs:

1. There is a memory leak when copy text from clipboard.

2. The whole notes file loads to memory that causes troubles with big files and troubles with the editor. It freezes when processing the really big file (tested on 50 mb markdown file).

3. It seems the app didn't unload notes from memory (e g when switched to another note). Previous should be saved and freed from memory after autosave period. Testing with 7 mb markdown file shows that data loaded but not unloaded when switched to another notes file. Memory consumption just raised from 25mb to 100mb.
   You could use this file for memory check
   Elephants killing markdown editors.zip

4. Nothing happens on crash with somehow made .conf file
   QOwnNotes.conf.zip

Possible fix for 2 and 3:

• Separately load text files into a fixed-size chunks (e.g 10 000 symbols for one chunk).
• Dynamically load/unload chunks to editor when needed. (It still could be stored in memory, to speed up )
• Change editor behaviour to calculate the viewport and pre-read next and previous chunks.
• Store undo/redo information on disk.
• And finally add file size restriction.

Fix for 4:

Store last file/session info and settings separately. To avoid config corruption.

system info

QOwnNotes Version 18.09.2
Qt Version 5.9.6
Desktop environment: KDE
gentoo-linux 4.14.67

[pbek]

Thank you for our detailed report! But most of the things like the editor, undo/redo and clipboard are handled by the Qt framework directly and you would need to file bugs there...

[pbek]

Can you make out what part of the config made your app not starting again?

[fodrice]

This one
[General]
allowOnlyOneAppInstance=true

[pbek]

Thank you, seems to be a problem of the external library https://github.com/pbek/QOwnNotes/tree/develop/src/libraries/singleapplication.

[fodrice]

It could help
https://stackoverflow.com/questions/5006547/qt-best-practice-for-a-single-instance-app-protection

[fodrice]

Checked memory again, every switch between big notes eats small amount of ram.

[pbek]

I am all open for Pull Request

[fodrice]

Added big markdown file I used for testing.
Elephants killing markdown editors.zip

[pbek]

18.09.3

• attempted to improve the single application mode
  • on certain occasions it was possible that the app wouldn't start a 2nd
    time even if it was closed, this should be fixed now

[pbek]

As for testing with 32MB markdown files... I wonder that you have even manage to open such a big text file in QOwnNotes... It took quite a while even on a SSD.

[pbek]

There now is a new release, could you please test it and report if it works for you?

[fodrice]

There now is a new release, could you please test it and report if it works for you?

I will test it tomorrow.

As for testing with 32MB markdown files

As I think the only way to improve this it to completely rewrite editor with MVC pattern. Render every line separately as view. And don't load the whole file but only beginning, load new file part when users need it (scrolls near it beginning or searching ... etc) and unloading unused parts. There will be lots of work... undo-redo, full-text search in file... and some more. Not Expected to be fixed soon.
But what really matters Is that after closing file memory remains in use.
Don't know c++ well. I dived into code a little and got a question. You load the whole text into database when file opened, don't you? Is it somehow unloaded when file is closed ?

[fodrice]

Single application mode now works fine for me. Tested latest appimage.

I wonder that you have even manage to open such a big text file in QOwnNotes

It's only 35mb it should be loaded to memory in less than 1 second...
It depends more on Processor, system bus, and memory speed than HDD speed.
And, obliviously, it depends on application. Cause app have to render that stuff. It opens in about 10 seconds on my system. (i5-2nd_gen,4cores,3.10GHz; Nvidia-970; gentoo-linux-4.14.67 kernel).
Meanwhile Kate (kde editor) opens the same file instantly on the same system.
But as I wrote, just add it to your "big things todo later" list.

[pbek]

You load the whole text into database when file opened, don't you? Is it somehow unloaded when file is closed ?

all note files are loaded into an in-memory sqlite database in the beginning. if the note is opened the note's text is loaded in a QPlainTextEdit

It's only 35mb it should be loaded to memory in less than 1 second...

I don't think that isn't the issue, I think the issue is editing so much text in a QPlainTextEdit

[pbek]

@fodrice did 18.09.3 bring any improvement with application locking after a crash?

[fodrice]

Yes. Now it works fine.

[pbek]

Thank you for testing! Since I cannot do much about the other stuff I will close this for now...