fall back os.fdatasync() to fsync() on POSIX systems without fdatasync() support

[gmelikov]

BPO	39640
Nosy	@gpshead, @ronaldoussoren, @serhiy-storchaka, @gmelikov, @cebtenzzre
PRs	bpo-39640: os.fdatasync: fall back to fsync() on POSIX systems withou… #18516

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-03-20.01:17:54.958>
created_at = <Date 2020-02-15.19:06:08.294>
labels = ['type-feature', 'expert-IO']
title = 'fall back os.fdatasync() to fsync() on POSIX systems without fdatasync() support'
updated_at = <Date 2022-03-20.01:17:54.957>
user = 'https://github.com/gmelikov'

bugs.python.org fields:

activity = <Date 2022-03-20.01:17:54.957>
actor = 'gregory.p.smith'
assignee = 'none'
closed = True
closed_date = <Date 2022-03-20.01:17:54.958>
closer = 'gregory.p.smith'
components = ['IO']
creation = <Date 2020-02-15.19:06:08.294>
creator = 'gmelikov'
dependencies = []
files = []
hgrepos = []
issue_num = 39640
keywords = ['patch']
message_count = 6.0
messages = ['362025', '362027', '362028', '362029', '375824', '415587']
nosy_count = 5.0
nosy_names = ['gregory.p.smith', 'ronaldoussoren', 'serhiy.storchaka', 'gmelikov', 'cebtenzzre']
pr_nums = ['18516']
priority = 'normal'
resolution = 'rejected'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue39640'
versions = []

[gmelikov]

POSIX fdatasync() is similar to fsync() but it tries not to sync non-needed metadata. If POSIX OS doesn't have it - it's safe to use fsync() (If we need to sync data to disk - we have to use one of these functions).

This change will help to run code with fdatasync() on MacOS without fallbacks in Python code.

I'll propose a PR soon.

[serhiy-storchaka]

fsync() may be slower than fdatasync(). There may be cases in which you prefer to not call fsync() or call it less often if POSIX fdatasync() is not available. To do this you should know whether Python fdatasync() calls POSIX fdatasync() or fsync(). The simplest way to provide this information to user is to not provide os.fdatasync() in the underlying system call is not available.

Note also that this change will not help to run code with fdatasync() on MacOS without fallbacks in Python code until you drop support of all Python versions older than 3.9.

[gmelikov]

If there is a way not to sync data - you should use neither fdatasync nor fsync.

So IMHO if someone wants to sync data and want to use fdatasync - I see the only way on MacOS is to use fsync().

Note also that this change will not help to run code with fdatasync() on MacOS without fallbacks in Python code until you drop support of all Python versions older than 3.9.

I agree, but I propose to make this change in 3.9 to get it somewhere in future.

[gmelikov]

I want to add that it's a usual practice:

• https://github.com/libuv/libuv/pull/1580/files/bdd987a9b4347164e31e22d2d5ce06fbb5ebc859
• https://rev.ng/gitlab/revng/qemu/commit/6f1953c4c14566d3303709869fd26201828b3ccf

[gmelikov]

PR rebased and ready to review.

[gpshead]

The os module provides a pretty low level simple shim over platform APIs. It is better for logic like this to live in a higher level application library rather than make big assumptions on the part of the user.

try:
    os.fdatasync(fd)
except Exception as err:
    logging.debug("fdatasync(fd) failed %s, falling back to fsync(fd)", err)
    os.fsync(fd)