-
-
Notifications
You must be signed in to change notification settings - Fork 31.7k
mailbox.Maildir re-reads directory too often #44299
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
[forwarded from http://bugs.debian.org/401395] Various functions in mailbox.Maildir call self._refresh, which always re-reads the cur and new directories with os.listdir. _refresh should stat each of the two directories first to see if they changed. This cuts processing time of a series of lookups down by a factor of the number of messages in the folder, a potentially large number. |
By stat()'ing the directories, do you mean checking the mtime? I think this isn't safe because of the limited resolution of mtime on filesystems; ext3 seems to have a 1-second resolution for mtime, for example. This means that _refresh() might read a directory, and if some other process adds or deletes a message in the same second, _refresh() couldn't detect the change. Is there some other property of directories that could be used for a more reliable check? The attached patch implements checking of mtime, but I don't recommend applying it; it causes the test suite in test_mailbox.py to break all over the place, because the process modifies mailboxes so quickly that the mtime check doesn't notice the process's own changes. I'll wait a bit for any alternative suggestion, and then close this bug as "won't fix". File Added: mailbox-mtime.patch |
Stray thought: would it help if the patch stored the (mtime - 1sec) instead of the mtime? Successive calls in the same second would then always re-read the directories, but once the clock ticks to the next second, re-reads would only occur if the directories have actually changed. The check would be 'if new_mtime > self._new_mtime' instead of '=='. Is this sort of mtime-based checking reliable on remote filesystems such as NFS? |
akuchling wrote:
Regardless of the resolution on any particular filesystem, stat and stat64
True. mailbox.Maildir's behavior of always representing the current contents The two solutions below (inotify and your suggested mtime-1 approach) would
On Linux, you could use inotify to get a notice when anything changes in the
Fair enough.
Please don't. The performance hit of repeatedly re-reading the directory akuchling:
Good idea. That would work fine as well, and I believe it would have Compared to inotify, this would have somewhat more overhead (but still far
This particular sort of checking, yes, I think so. The times do not Thank you for looking at this problem, |
Updated version of the patch that only stores the current time -1sec, |
Committed to trunk in rev. 72213. Committed to py3k in rev. 72228. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: