mmap should expose madvise()

[pitrou]

BPO	32941
Nosy	@ronaldoussoren, @pitrou, @larryhastings, @giampaolo, @ned-deily, @MojoVampire, @miss-islington
PRs	bpo-32941: Add madvise() for mmap objects #6172 bpo-32941: Fix test_madvise failure when page size >= 8kiB #13596

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-05-27.16:48:56.032>
created_at = <Date 2018-02-24.21:15:23.475>
labels = ['3.8', 'type-feature', 'library']
title = 'mmap should expose madvise()'
updated_at = <Date 2019-05-27.17:57:26.307>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2019-05-27.17:57:26.307>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2019-05-27.16:48:56.032>
closer = 'pitrou'
components = ['Library (Lib)']
creation = <Date 2018-02-24.21:15:23.475>
creator = 'pitrou'
dependencies = []
files = []
hgrepos = []
issue_num = 32941
keywords = ['patch']
message_count = 6.0
messages = ['312758', '312759', '339375', '343659', '343660', '343667']
nosy_count = 8.0
nosy_names = ['ronaldoussoren', 'pitrou', 'larry', 'giampaolo.rodola', 'ned.deily', 'neologix', 'josh.r', 'miss-islington']
pr_nums = ['6172', '13596']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue32941'
versions = ['Python 3.8']

[pitrou]

On POSIX, mmap objects could expose a method wrapping the madvise() library call. I suggest the following API

  mmap_object.madvise(option[, start[, length]])

If omitted, *start* and *length* would span the whole memory area described by the mmap object. *option* must be a recognized OS option for the madvise() library call.

The mmap module would expose the various MADV_* options available on the current platform.

Open question: should we expose madvise() or posix_madvise()? (these are two different calls, at least on Linux) posix_madvise() is arguably more portable, but madvise() is much more powerful, so I'd lean towards madvise().

[pitrou]

For information:

• macOS manpage for madvise():
  http://www.manpages.info/macosx/madvise.2.html
• FreeBSD manpage for madvise():
  https://www.freebsd.org/cgi/man.cgi?query=madvise&sektion=2

[MojoVampire]

It might be nice to expose a more limited API to prefetch memory in bulk while we're at it. Windows 8 and higher offers PrefetchVirtualMemory ( https://docs.microsoft.com/en-us/windows/desktop/api/memoryapi/nf-memoryapi-prefetchvirtualmemory ) which fills roughly the same niche as madvise/posix_madvise with the WILLNEED hint.

I kept meaning to file an issue to suggest this, but kept getting hung up on how to make it as portable as possible and as powerful as possible, and where to implement it.

It could go on mmap, but it's also useful for just about any contiguous buffer-supporting object, so it almost seems like adding an os.madvise (and/or os.prefetch) would make more sense than specifically tying it to the mmap module.

[pitrou]

New changeset 02db696 by Antoine Pitrou (Zackery Spytz) in branch 'master':
bpo-32941: Add madvise() for mmap objects (GH-6172)
02db696

[pitrou]

Josh, sorry, I hadn't seen your message.

Those are low-levels operations, so I don't know if it makes sense to implement madvise() in terms of PrefetchVirtualMemory(), or expose a separate wrapper to PrefetchVirtualMemory().

One complication is that we currently don't expose public wrappers over Windows API functions (there's a private _winapi module).

[miss-islington]

New changeset 695b1dd by Miss Islington (bot) (Antoine Pitrou) in branch 'master':
bpo-32941: Fix test_madvise failure when page size >= 8kiB (GH-13596)
695b1dd