New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hickup/freeze when notes save on Mac #291

Closed
minthemiddle opened this Issue Aug 15, 2016 · 31 comments

Comments

Projects
None yet
4 participants
@minthemiddle

minthemiddle commented Aug 15, 2016

When QON autosaves note changes on Mac, there is a noticeable hickup and freeze. Then you see the notification that the note was edited outside of QON. You can turn that notification off as there are no visible modifications that Mac executes, but the freeze remains. I increased the auto-save time to 30 secs to get a smooth editing experience, but that is just a workaround (which is also a bit risky).

First I thought it was due to Dropbox as described in #171 but I tested it now with a regular folder and have the very same problem. It must be something with the way Mac saves files or metadata.

(Running latest MacOSX and QON)

@pbek

This comment has been minimized.

Owner

pbek commented Aug 15, 2016

Often, not always, files in the note folder get modified externally when a note is saved (metadata could be a reason), this causes the all notes in the note folder being read again (because QOwnNotes doesn't know what was changed).

If you have a mac with a spinning disk and a lot of notes this takes some time...

I've hunting this for a long time and already tried several workarounds.
Do you have any ideas or suggestions?

@minthemiddle

This comment has been minimized.

minthemiddle commented Aug 15, 2016

Ok, so it's not just me. I can do some research on this as well.

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Aug 16, 2016

If I remember correct the app uses inotify events to get external modification. This events contain the modified file name, at least on Linux. So the app can know exactly what file was changed externally. Why does it update the whole folder?

@pbek

This comment has been minimized.

Owner

pbek commented Aug 16, 2016

To get a file specific notification you have to watch that specific file. And we can't watch a file we don't know. If we don't know the file the notification is for the whole folder.

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Aug 16, 2016

On Linux I watch a directory and get all events within it with file name and event type. I've tested it and it works very good in my script.

I'll read some more docs about QFileSystemWatcher. Maybe it can be make to work on file level for the app.

@pbek

This comment has been minimized.

Owner

pbek commented Aug 16, 2016

QFileSystemWatcher is what we use here. It watches files and directories that you add to it.

@pbek pbek referenced this issue Aug 17, 2016

Closed

Global note search #186

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Aug 17, 2016

As far as I've understand the app uses QFileSystemWatcher only on directory level watching note root directory and it's sub-directories. To get it working on file level we need to feed it file names and wait for some fileChanged signals. Also we need some kind of buffer.

Maybe it can be done by something like that.

  1. Add note directory to watcher
  2. Get list of all files there, store their names and content into array. Then add all files names to watcher.
  3. On directoryChanged signal get the new file list and compare their names against names in array. Getting list of file names should be quite fast.
  4. Read only new files which are missing and add them to array. Add this new files to watcher too. Remove deleted files from array.
  5. Update from array without reading all the files.
  6. On fileChanged signal read only the changed file and update it's content in array. Then update from array.
@pbek

This comment has been minimized.

Owner

pbek commented Aug 17, 2016

Currently not all note files are watched to not get over the open files limit on various operating systems. Other that that it might work...

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Aug 18, 2016

As far as I know the lowest limit is for FreeBSD's kqueue and it's 1024 paths. I don' know what QFileSystemWatcher use on OSX - old kqueue or fsevents.
Next is Linux which typically has 8192 or more.

To make sure the app won't exceed the limit you can check number of elements in array before adding it all to watcher.

If it's more than 256 or 512 you can fallback to old directory only mode. I doubt many users have more than 512 notes in a single folder.

@pbek

This comment has been minimized.

Owner

pbek commented Aug 18, 2016

I've set my limit to millions under Linux and still had some files not watched until I decided to only watch the newest 200 notes...
But no worries, I will compare file sizes and modification dates of all notes...

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Aug 18, 2016

Good idea. Simple and effective.

@pbek

This comment has been minimized.

Owner

pbek commented Aug 18, 2016

Effective, maybe. Simple, not at all, because since the note sub folders will be re-created too in the process none of the notes know which sub folder they should belong to...

@pbek pbek added this to the 16.08.13 milestone Aug 18, 2016

@pbek

This comment has been minimized.

Owner

pbek commented Aug 18, 2016

16.08.13

  • the reloading speed of the note folder and note subfolders has
    been dramatically improved
    • when a reload is triggered now only modified notes and folders
      will be loaded again
    • a reload happens for example if a file, that wasn't watched by
      QOwnNotes was modified outside the application
@pbek

This comment has been minimized.

Owner

pbek commented Aug 18, 2016

There now is a new release, could you please test it and report if the new features work for you?

@pbek

This comment has been minimized.

Owner

pbek commented Aug 19, 2016

16.08.14

  • when the notes are reloaded the note list and the note subfolder tree
    will now only be refreshed in the user interface if a note or a note
    subfolder was really modified
@pbek

This comment has been minimized.

Owner

pbek commented Aug 19, 2016

has anyone tested that yet?

@Maboroshy

This comment has been minimized.

Contributor

Maboroshy commented Aug 19, 2016

I will be able to test it no earlier than Sunday.

@pbek

This comment has been minimized.

Owner

pbek commented Aug 19, 2016

great ;)
@minthemiddle ?

@minthemiddle

This comment has been minimized.

minthemiddle commented Aug 19, 2016

It certainly is faster on Mac, fast enough with 16.08.14 to reduce my saving intervall back to 5 seconds. I have different notes folders and I still see quite a difference in the time it takes to save: project folders with about 50 notes on my harddrive = no more hickup. One note folder with about 2000 notes (my second brain) in a Dropbox folder. Unfortunately I can still sense a hickup here. I have no idea what is causing the lag here: Dropbox or the amount of notes.

EDIT: Remove non-helping video

@pbek

This comment has been minimized.

Owner

pbek commented Aug 19, 2016

Thank you for testing, so where exactly does the delay occur? I can't really make it out in the gif you have posted. ;)
Is there any useful information in the log dialog?

@minthemiddle

This comment has been minimized.

minthemiddle commented Aug 19, 2016

It's actually the time lag between switching between the two lines (with blue background) on top. My log says something like this. The time I changed. The lag is probably within that second between folder and note modification.

[16:35:17] [status] Notizordner wurde extern verändert (Folder externally modified)
[16:35:18] [status] Notiz wurde extern verändert: /Users/x/Dropbox/x/138.txt (Note externally modified)
@pbek

This comment has been minimized.

Owner

pbek commented Aug 19, 2016

If the folder was modified externally all 2000 files have to be scanned. That might take some time on a spinning disk. ;)

@pbek pbek closed this Aug 19, 2016

@pbek

This comment has been minimized.

Owner

pbek commented Aug 19, 2016

@minthemiddle, btw., do the shortcuts (like cmd + N) currently work for you under OS X and do you see them in the main menu?

@minthemiddle

This comment has been minimized.

minthemiddle commented Aug 22, 2016

@pbek can check whether shortcuts work or not later today. I wonder whether an option like Ignore all external folder changes (similar to the Ignore all external file changes) would help to speed up folders with many notes on OSX. Of course at everyone's own risk, of course the *.sql file should still be watched for changes but maybe not the rest. Because, even if OSX somehow magically changes things, for sure this is not relevant changes and non-breaking. I would certainly risk it for higher performance.

@pbek

This comment has been minimized.

Owner

pbek commented Aug 22, 2016

@minthemiddle, I will think about that... It's not an easy topic.

@pbek

This comment has been minimized.

Owner

pbek commented Aug 22, 2016

16.08.17

  • you can now turn off the refreshing of the note list on external note
    folder changes completely in the general settings if for example you
    have so many notes, that even the checking for file-time and file-size
    changes is taking too long
    • use this at your own risk!

@minthemiddle, the note database should be handled by sqlite, maybe you can test that while the new checkbox is turned off/on in the new release.

@pbek pbek added this to the 16.08.17 milestone Aug 22, 2016

@pbek pbek removed this from the 16.08.13 milestone Aug 22, 2016

@pbek

This comment has been minimized.

Owner

pbek commented Aug 22, 2016

There now is a new release, could you please test it and report if the new features work for you?

@minthemiddle

This comment has been minimized.

minthemiddle commented Aug 24, 2016

Just tested it on OSX 10.11.6 (15G31) and it runs just fine. It really speed up my folder with 2000+ notes. And I have not experienced any problems thus far. Perfect and rather risk free if you really just work in QON and change one note after another.

@pbek

This comment has been minimized.

Owner

pbek commented Aug 24, 2016

thank you for testing, @minthemiddle!

@DivineDominion

This comment has been minimized.

DivineDominion commented Dec 4, 2016

I tested the latest development release and on my SSD MacMini editing one of 3800 notes in the archive takes a bit of time: http://d.pr/v/Ij97 (You don't see the cursor change to beachball when I wait for the app to respond again; there are 2 delays: when I switch to another note in the beginning and during typing in the 2nd note)

With the "ignore external folder changes" settings turned on, the delay goes away when I edit in favor of a message box saying the note was modified externally. The notes are not stored in Dropbox and there's no other text editor using the directory, so I guess QOwnNotes's saving triggers a redundant event. (Comparing modification dates could really help.) When I tick "ignore modifications" for the current note, performance is ace. But this comes at the obvious cost of not being able to access the notes folder from 2 devices/sync via Dropbox.

If you can conditionally switch file system monitoring: FSEvents on macOS is very fast and can report changes to files within a folder (1 level deep).

I don't know my way around Qt. Maybe you can copy the approach of node.js's abstraction of file system events from their source. Here's a popular library that unifies file system events cross-platform: https://www.npmjs.com/package/chokidar

@pbek

This comment has been minimized.

Owner

pbek commented Dec 4, 2016

Thank you for testing, the question is what causes the external folder changes under macOS... I haven't found a way to find that out yet.
And I don't fancy the idea of integrating native macOS code and reimplement the same thing a 2nd time. It's pretty hairy already.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment