test_os: test_get_set_inheritable_o_path() failed on AMD64 FreeBSD Shared 3.x

[vstinner]

BPO	44849
Nosy	@vstinner, @ambv, @koobs, @pablogsal, @miss-islington, @erlend-aasland
PRs	bpo-44849: Fix os.set_inheritable() on FreeBSD 14 with O_PATH #27623 [3.9] bpo-44849: Fix os.set_inheritable() on FreeBSD 14 with O_PATH (GH-27623) #27630 [3.10] bpo-44849: Fix os.set_inheritable() on FreeBSD 14 with O_PATH (GH-27623) #27631 [3.8] bpo-44849: Fix os.set_inheritable() on FreeBSD 14 with O_PATH (GH-27623) #28978

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-08-06.15:14:21.237>
created_at = <Date 2021-08-06.10:40:46.152>
labels = ['tests', '3.9', '3.10', '3.11']
title = 'test_os: test_get_set_inheritable_o_path() failed on AMD64 FreeBSD Shared 3.x'
updated_at = <Date 2021-10-19.19:15:33.700>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-10-19.19:15:33.700>
actor = 'lukasz.langa'
assignee = 'none'
closed = True
closed_date = <Date 2021-08-06.15:14:21.237>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2021-08-06.10:40:46.152>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 44849
keywords = ['patch']
message_count = 11.0
messages = ['399065', '399067', '399069', '399085', '399088', '399089', '399167', '399168', '399169', '399264', '404340']
nosy_count = 6.0
nosy_names = ['vstinner', 'lukasz.langa', 'koobs', 'pablogsal', 'miss-islington', 'erlendaasland']
pr_nums = ['27623', '27630', '27631', '28978']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue44849'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[vstinner]

Since build 655 (commit 6871fd0), test_os failed on AMD64 FreeBSD Shared 3.x:
https://buildbot.python.org/all/#/builders/483/builds/655

======================================================================
ERROR: test_get_set_inheritable_o_path (test.test_os.FDInheritanceTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_os.py", line 3898, in test_get_set_inheritable_o_path
    os.set_inheritable(fd, True)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 9] Bad file descriptor

[vstinner]

test_get_set_inheritable_o_path() also fails on:

• AMD64 FreeBSD Shared 3.10
• AMD64 FreeBSD Shared 3.9

It smells more like a FreeBSD system update rather than a Python regression.

[vstinner]

On my FreeBSD 13 VM, the test is skipped:

vstinner@freebsd$ ./python -m test -v test_os -m test_get_set_inheritable_o_path
== CPython 3.11.0a0 (heads/main:938e84b4fa, Aug 6 2021, 10:46:32) [Clang 11.0.1 (git@github.com:llvm/llvm-project.git llvmorg-11.0.1-0-g43ff75f2c3
== FreeBSD-13.0-RELEASE-amd64-64bit-ELF little-endian
(...)
test_get_set_inheritable_o_path (test.test_os.FDInheritanceTests) ... skipped 'need os.O_PATH'

It seems like O_PATH was introduced in FreeBSD 14 CURRENT, but the test fails.

Code of the test:

@unittest.skipUnless(hasattr(os, 'O_PATH'), "need os.O_PATH")
def test_get_set_inheritable_o_path(self):
    fd = os.open(__file__, os.O_PATH)
    self.addCleanup(os.close, fd)
    self.assertEqual(os.get_inheritable(fd), False)

        os.set_inheritable(fd, True)  # <==== OSError: [Errno 9] Bad file descriptor
        self.assertEqual(os.get_inheritable(fd), True)

        os.set_inheritable(fd, False)
        self.assertEqual(os.get_inheritable(fd), False)

os.set_inheritable() calls _Py_set_inheritable() whichs has multiple implementations:

• ioctl(fd, FIONCLEX, NULL)
• fcntl(fd, F_GETFD) + fcntl(fd, F_SETFD, new_flags)

Linux has a special code path:

#ifdef __linux__
        if (errno == EBADF) {
            // On Linux, ioctl(FIOCLEX) will fail with EBADF for O_PATH file descriptors
            // Fall through to the fcntl() path
        }
        else
#endif

Maybe we should just enable now this code path on FreeBSD as well.

== test.pythoninto of the last successful build 654 ==

datetime.datetime.now: 2021-08-01 01:15:06.918625

os.uname: posix.uname_result(sysname='FreeBSD', nodename='140-CURRENT-amd64-564d', release='14.0-CURRENT', version='FreeBSD 14.0-CURRENT #14 main-n245176-8742817ba62: Tue Mar 2 03:48:10 UTC 2021 root@140-CURRENT-amd64:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG', machine='amd64')

== test.pythoninto of the failed build 655 ==

datetime.datetime.now: 2021-08-04 09:56:34.287999

os.uname: posix.uname_result(sysname='FreeBSD', nodename='140-CURRENT-amd64-564d', release='14.0-CURRENT', version='FreeBSD 14.0-CURRENT #0 main-n248401-de0c7fbe280: Mon Aug 2 02:42:31 UTC 2021 root@140-CURRENT-amd64-564d:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG', machine='amd64')

[vstinner]

New changeset c24896c by Victor Stinner in branch 'main':
bpo-44849: Fix os.set_inheritable() on FreeBSD 14 with O_PATH (GH-27623)
c24896c

[miss-islington]

New changeset 2ae2235 by Miss Islington (bot) in branch '3.10':
bpo-44849: Fix os.set_inheritable() on FreeBSD 14 with O_PATH (GH-27623)
2ae2235

[miss-islington]

New changeset 791c28a by Miss Islington (bot) in branch '3.9':
bpo-44849: Fix os.set_inheritable() on FreeBSD 14 with O_PATH (GH-27623)
791c28a

[koobs]

There was a recent worker update to latest CURRENT (14). I'll try to identify recent changes or changesets that may have affected behaviour and reference them here

[koobs]

O_PATH support was added in FreeBSD CURRENT April 15:

https://reviews.freebsd.org/D29323?id=86257

freebsd/freebsd-src@8d9ed17

@victor If the failure in this tests suggests either a bug in our implementation, or an incompatibility, please let me know/clarify so I can get our implementation polished

[koobs]

Reference for Linux behaviour:

filedesc: don't use ioctl(FIOCLEX) on Linux bpo-62425
rust-lang/rust#62425

Looks like we match the behaviour intentionally (or otherwise).

[vstinner]

AMD64 FreeBSD Shared 3.x is back to green: test_os.test_get_set_inheritable_o_path() now pass on FreeBSD CURRENT ;-) My fix works as expected. I simply used the same fix than for Linux.

[ambv]

New changeset 67e10be by Miss Islington (bot) in branch '3.8':
bpo-44849: Fix os.set_inheritable() on FreeBSD 14 with O_PATH (GH-27623) (GH-28978)
67e10be