O_DIRECT read fails with 4K mmap buffer

[yoyoyopcp]

BPO	38167
Nosy	@pitrou, @MojoVampire, @yoyoyopcp
PRs	bpo-38167 Allow for 4K O_DIRECT reads. #16130 bpo-38167 Allow for 4K O_DIRECT reads. #16131

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-10-11.02:40:05.444>
created_at = <Date 2019-09-13.23:28:40.470>
labels = ['type-bug', '3.8', '3.9', 'expert-IO', '3.7', 'invalid']
title = 'O_DIRECT read fails with 4K mmap buffer'
updated_at = <Date 2019-10-11.02:40:05.442>
user = 'https://github.com/yoyoyopcp'

bugs.python.org fields:

activity = <Date 2019-10-11.02:40:05.442>
actor = 'josh.r'
assignee = 'none'
closed = True
closed_date = <Date 2019-10-11.02:40:05.444>
closer = 'josh.r'
components = ['IO']
creation = <Date 2019-09-13.23:28:40.470>
creator = 'yoyoyopcp'
dependencies = []
files = []
hgrepos = []
issue_num = 38167
keywords = ['patch']
message_count = 9.0
messages = ['352397', '352403', '352404', '353942', '353943', '353951', '353984', '353988', '354417']
nosy_count = 3.0
nosy_names = ['pitrou', 'josh.r', 'yoyoyopcp']
pr_nums = ['16130', '16131']
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue38167'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9']

[yoyoyopcp]

The following Python3 script fails.

import mmap
import os

fd = os.open(path_to_file, os.O_DIRECT | os.O_RDWR)
fo = os.fdopen(fd, 'rb+')
m = mmap.mmap(-1, 4096)
fo.readinto(m)

But it worked for Python2. It also works for any other multiple of 4K. For example:

m = mmap.mmap(-1, 8192)
fo.readinto(m)

Is fine!

[MojoVampire]

Works just fine for me on 3.7.3 on Ubuntu, reading 4096 bytes. How is it failing for you? Is an exception raised?

It does seem faintly dangerous to explicitly use O_DIRECT when you're wrapping it in a buffered reader that doesn't know it has to read in units matching the minimum block size (file system dependent on older kernels, 512 bytes in Linux kernel 2.6+); BufferedIOBase.readinto is explicitly documented to potentially issue multiple read calls (readinto1 guarantees it won't do that at least).

[yoyoyopcp]

This is the platform that I'm working on as well as the failure. I have a review out for a fix.

# uname -a
Linux init129-13 3.10.0-957.el7.x86_64 x86_64 x86_64 x86_64 GNU/Linux
# python3.7 directread.py
Traceback (most recent call last):
  File "small.py", line 7, in <module>
    fo.readinto(m)
OSError: [Errno 22] Invalid argument

[pitrou]

I agree with Josh. If you want to use O_DIRECT, use an unbuffered file object and be sure to issue reads of the right size.

[pitrou]

Also I'm curious: why are you using O_DIRECT, and furthermore, why are you using it to read into mmap'ed memory?

[yoyoyopcp]

I agree with Josh. If you want to use O_DIRECT, use an unbuffered file object and be sure to issue reads of the right size.

I do not believe an unbuffered file uses O_DIRECT. This is why I use os.open(fpath, os.O_DIRECT).

Also I'm curious: why are you using O_DIRECT, and furthermore, why are you using it to read into mmap'ed memory?

I am testing a storage device and must use O_DIRECT to avoid the kernel's cache. I am using mmap because it was the simplest way to get a page-aligned memory buffer, which is required for direct IO.

I believe that this is a bug regardless of the use of mmap, especially considering that this worked in Python 2. I believe the fix I have sent out for review addresses it adequately.

[MojoVampire]

I do not believe an unbuffered file uses O_DIRECT. This is why I use os.open(fpath, os.O_DIRECT).

Problem is you follow it with:

fo = os.fdopen(fd, 'rb+')

which introduces a Python level of buffering around the kernel unbuffered file descriptor. You'd need to pass buffering=0 to make os.fdopen avoid returning a buffered file object, making it:

fo = os.fdopen(fd, 'rb+', buffering=0)

[yoyoyopcp]

Problem is you follow it with:

fo = os.fdopen(fd, 'rb+')

which introduces a Python level of buffering around the kernel unbuffered file descriptor. You'd need to pass buffering=0 to make os.fdopen avoid returning a buffered file object, making it:

fo = os.fdopen(fd, 'rb+', buffering=0)

You are absolutely right! This fixed the issue. So... is this not a bug, then? Should I discard my patch?

[MojoVampire]

Yeah, not a bug. The I/O subsystem was substantially rewritten between Python 2 and Python 3, so you sometimes need to be more explicit about things like buffering, but as you note, once the buffering is correct, the code works; there's nothing to fix.