Please, do not lock the log file

[evandrocoan]

I use the Python logging module RotatingFileHandler https://docs.python.org/2/library/logging.handlers.html#rotatingfilehandler

Which renames the log file file.log to file.log.1 when it reaches a maximum size like 50MB, however if I am viewing the log file with this tool, the operation fails because the file is locked by

[variar]

The issue is explained upstream: nickbnf#109. File is read from disk for each search and for redrawing. There are performance issues on windows if file is closed and opened each time. File is opened using QFile::open(QIODevice::ReadOnly) and I don't know if one can manage locking that way

[shvez]

In general Windows allows to open file with delete access. one needs to specify FILE_SHARE_DELETE for CreateFile api call

[stolsvik]

(Unrelated) @evandrocoan Which tool are your screenshot from?

[evandrocoan]

https://jetscreenshot.com/, but it is a paid app. For free there is also http://www.donationcoder.com/software/mouser/popular-apps/screenshot-captor

[shvez]

@variar did you fix this or just closed?

[variar]

@shvez I hope that the problem is fixed. I've changed file opening on Windows to use native CreateFile function with shareMode = FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE. This should allow to rename files that are opened in klogg. If you try to delete file it will get removed when it is closed in klogg.

This commit should be in build >= 19.1.0.322 (https://github.com/variar/klogg/releases/tag/continuous-win). It would be great if someone could test this because I don't use follow file feature and changing log files in my daily job.

[shvez]

UPD:
@variar I did try and should admit that it does not work as expected :(. Yes, it works as you described but that is something different from what baretail does. Renaming also does not work - access denied :( So we definitely need to use some other trick.

Thanks a lot for this try to make us happy.

[variar]

Access denied is generated when one is trying to overwrite file opened in klogg. However, renaming opened file works fine. So one could first rename opened file and after that create a new one with old name.

In nickbnf#109 there was a suggestion to add option to actually keep file closed and open it every time some data is required. I'll investigate this solution.

[variar]

I added option to keep file closed (see 975c3d7). Can't add ui for it right now, but it should appear in configuration file after closing new version of klogg. After setting perf.keepFileClosed=true in configuration file and reloading klogg it should keep file open only during indexing or reading data. Configuration file is located in %AppData%\klogg\klogg.ini (or klogg.conf near executable for portable version)

The fix is in 19.1.0.233 build.

There could be issues with file changing on disk during index being rebuilt or search in progress. Need more testing in real-life scenarios.

[variar]

Since version 19.1.0.235 there is an option in advanced settings to keep file closed as much as possible

[shvez]

@variar That what I wanted!!. Thank you very much

The only bug I found is a failure after renaming. If I delete opened file, its page cleaned up. But if I rename the file, then its content still on screen, and I get trash if try to search using window below

[variar]

Great news @shvez, thanks for testing this.

Looks like right now if new file is larger than old one, then only partial index operation is performed for "appended lines", and everything is trashed. Should not happen when overwriting old file with smaller one (that should be the main scenario for rolling log files?)

I'll try to figure out some way to check if new file is really not the same as old one and do full reload (inode checking?)

[shvez]

In my case, I just renamed a file to something else. Behavior of the tool should be same like file would be deleted

[variar]

Issue with simple renaming of opened file should be fixed in 19.1.0.237.

Overwriting file with larger one completely unrelated file needs further investigation.

[shvez]

thank you will check it out soon

[wiz0u]

Tested klogg-19.01.0.337-x64-portable.zip : 'Enable Polling' doesn't seem to work. :(
I'm monitoring a log on a network share updated almost every second. Klogg doesn't detect any update to the log after it has been opened.

[variar]

File monitoring on network shares has not been tested. Seems similar to nickbnf#214. Looks like file monitoring api works with network drives, but only if the remote server supports the functionality. What system are you using as network share server?

[variar]

@wiz0u I've moved problems with file monitoring and polling to more appropriate issue.

This issue should stay for problems with klogg locking files and preventing other applications access them.

[wiz0u]

Ok for moving to a separate issue, but in my opinion the locking issue made the most sense on network shares where multiple users can view the file through glogg/klogg and possibly prevent daily rolling/archiving/deleting. It is worse than a purely local issue because you cannot tell all users to close their log viewer before going home.

[variar]

I've a feeling that polling could be locking file for brief period (to get current size and modification time). This might still be breaking rolling logs in rare cases. Randomizing polling time so file is not accessed on "round" time (e.g. at the start of a minute) could help to further reduce locking possibility.

[variar]

On the other hand polling timer start time is already random. Will close for now.