"mmap.flush()" is always synchronous, hurting performance

[jcea]

BPO	18816
Nosy	@jcea, @MojoVampire

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2013-08-23.04:55:08.491>
labels = ['extension-modules', 'easy', 'type-feature']
title = '"mmap.flush()" is always synchronous, hurting performance'
updated_at = <Date 2014-03-07.02:57:27.168>
user = 'https://github.com/jcea'

bugs.python.org fields:

activity = <Date 2014-03-07.02:57:27.168>
actor = 'josh.r'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Extension Modules']
creation = <Date 2013-08-23.04:55:08.491>
creator = 'jcea'
dependencies = []
files = []
hgrepos = []
issue_num = 18816
keywords = ['easy']
message_count = 3.0
messages = ['195941', '195948', '195971']
nosy_count = 3.0
nosy_names = ['jcea', 'neologix', 'josh.r']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue18816'
versions = ['Python 3.4']

[jcea]

Currently, "mmap.flush()" does a synchronous write to the backend file. The call will wait until data is actually flushed to disk, because internally it is doing a "msync(MS_SYNC)".

But the value of "mmap.flush()" is to synchronize file and memory. You don't need a synchronous write in the general case.

I propose to add an optional keyword parameter with default value "SYNC" (compatibility) but that can be "ASYNC", "INVALIDATE" (can be "SYNC|INVALIDATE" and "ASYNC|INVALIDATE" too).

I am talking about UNIX MMAP. No idea about Windows.

Check "man msync" for useful cases.

[neologix]

I propose to add an optional keyword parameter with default value "SYNC" (compatibility) but that can be "ASYNC", "INVALIDATE" (can be "SYNC|INVALIDATE" and "ASYNC|INVALIDATE" too).

AFAICT it's mostly useless on a modern OS.
MS_INVALIDATE is a no-op on systems with merged VM-buffer cache, i.e.
it's not needed for mmap() to reflect write() and vice-versa.

So nothing's normally needed to "synchronize file and memory".

As for MS_ASYNC, it actually doesn't do anything at all on recent OS,
e.g. it's a no-op on Linux since a couple years, since modified pages
will be written back as part of the normal writeback process.

The only thing a user might actually need for an mmap object is to
make sure data is actually committed to disk, and MS_SYNC covers this.

See e.g. this post by Andrew Morton:
http://thread.gmane.org/gmane.linux.kernel/1312660

[jcea]

Depending of a concrete OS implementation is not good. Linux is not the only OS out there, and I have very old machines in production yet:

"""
# uname -a
Linux colquide.XXXX.es 2.4.37 #4 Fri Dec 12 01:10:45 CET 2008 i686 unknown
"""

I have been hit by the VM/file cache split in the past. Portability is important.

Anyway, the Python "mmap" manual says that "mmap.flush()" is needed to be sure that you are not going to "lose" changes you made in the mmap. On "modern" OSs it is not actually needed, as you say, and the performance hit is important enough for me to investigate and write this enhancement proposal :).