Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Persistent open files leak fix #2
This fixes a performance problem with (at least) incremental backups. My incr backups were slower the longer it was since the last reboot. Investigation showed the delay was proportional to the size of open_files.dat, with the indexing phase taking over an hour a week after a reboot instead of ~4 minutes seen immediately after a reboot. The problem is that open_files.dat, and the in-memory representation in 'open_write_files', contains references to many files which are no longer open. On my system, there should be around 300 open files, but after a week, we think there are about 90K open files.
Code in ChangeJournalWatcher.cpp failed to remove many files from open_write_files. Most of that was because USN_REASON_RENAME_OLD_NAME should be treated like USN_REASON_CLOSE when handling a normal file, removing the file from open_write_files.
The other leak was a timing problem - if multiple USN change journal records for a single file are split across multiple calls to ChangeJournalWatcher::update(), the file might get added to open_write_files after the first update() call, but the USN records for the file in the second update() could be handled completely with local_open_write_files, not noticing that the file had already been added to open_write_files and should be removed on a close journal record. Fix is, when adding a file to local_open_write_files, check whether the file is already in open_write_files, and if yes, remove it from both versions of the map when encountering a close journal record.
Including a compile minor changes. First, PrettyPrintTime was returning an empty string for times less than a second. Second, ChangeJournalWatcher::update_longliving should run its loops within a BeginTransaction/EndTransaction so the updates to the mfiles table are batched.