Drop Solaris support

[vstinner]

BPO	42173
Nosy	@rhettinger, @terryjreedy, @jcea, @vstinner, @giampaolo, @jstasiak, @pablogsal, @isidentical, @kulikjak, @gaige, @Jongy, @wiedi, @jclulow, @citrus-it, @alanc, @brett, @robertdfrench, @tbalbers, @mariuspana, @siepkes, @cgrzemba, @marcheschi, @ab, @jurajlutter, @YanChii, @jgardner100, @gwr
PRs	bpo-42173: WIP: Drop Solaris support #23002
Files	Oracle_Solaris_full_test.txt Oracle_Solaris_detailed_test.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-11-16.17:05:24.696>
created_at = <Date 2020-10-27.15:59:08.312>
labels = ['interpreter-core', 'invalid']
title = 'Drop Solaris support'
updated_at = <Date 2020-11-16.17:05:24.695>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2020-11-16.17:05:24.695>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-11-16.17:05:24.696>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2020-10-27.15:59:08.312>
creator = 'vstinner'
dependencies = []
files = ['49558', '49559']
hgrepos = []
issue_num = 42173
keywords = ['patch']
message_count = 49.0
messages = ['379767', '379886', '379887', '379888', '379890', '379891', '379893', '379894', '379895', '379897', '379899', '379900', '379901', '379906', '379910', '379915', '379924', '379925', '379926', '379927', '379933', '379937', '379938', '379942', '379946', '379955', '379956', '379959', '379961', '379962', '379963', '379991', '379992', '379993', '379994', '380006', '380008', '380080', '380196', '380203', '380205', '380210', '380241', '380256', '380257', '380455', '380456', '380457', '381122']
nosy_count = 37.0
nosy_names = ['rhettinger', 'terry.reedy', 'jcea', 'vstinner', 'giampaolo.rodola', 'jstasiak', 'pablogsal', 'BTaskaya', 'kulikjak', 'tropikhajma@gmail.com', 'gaige', 'Yonatan Goldschmidt', 'wiedi', 'mimi.vx', 'jclulow', 'omnios', 'pfirmsto', 'alanc', 'brett3', 'robertdfrench', 'jm650', 'tbalbers', 'mariuspana', 'siepkes', 'cgrzemba', 'marcheschi', 'abarbu', 'predrag.zecevic.1961', 'jorge.schrauwen', 'olafbohlen', 'otis', 'yanchii', 'michael.laspina', 'ijdt.editor', 'jomasoftmarcel', 'jgardner100', 'gwr']
pr_nums = ['23002']
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue42173'
versions = []

[vstinner]

In past years, very few Python core developers took care of the Solaris support. The latest significant Solaris enhancement was in 2018 with the addition of a new Solaris st_fstype attribute to os.stat(), in Python 3.7, by Jesús Cea Avión (bpo-32659).

Another recent Solaris-related change is bpo-41687: "Fix implementation of sendfile to be compatible with Solaris".

On the Oracle side, there was an article about Python on Solaris in 2019:
"Future of Python on Solaris" by Darren Moffat
https://blogs.oracle.com/solaris/future-of-python-on-solaris

I now that Giampaolo Rodola cares of the Solaris support for his psutil project:
https://github.com/giampaolo/psutil/

It's unclear to me if Oracle still plans to maintain Solaris. The latest release was published 2 years ago. The Wikipedia article says: "While Oracle did have a large layoff of Solaris development engineering staff, development continues today of which Solaris 11.4 was released in 2018."
https://en.wikipedia.org/wiki/Solaris_(operating_system)

There is currently 25 open issues which could be closed if we drop the Solaris support.

See also the PEP-11 for the process.

I asked asked 4 years ago (in 2016) if Solaris support should/can be removed:
"OpenIndiana and Solaris support"
https://mail.python.org/archives/list/python-dev@python.org/message/NOT2RORSNX72ZLUHK2UUGBD4GTPNKBUS/

Python has no longer buildbot workers running on Solaris or a Solaris variant (OpenIndiana, Illumos, etc.).

I propose to drop the Solaris support in Python to reduce the maintenance burden.

Python has a few features specific to Solaris:

• st_fstype attribute of os.stat()
• select.devpoll and selectors.DevpollSelector
• time.CLOCK_HIGHRES constant
• stat.S_ISDOOR()
• tarfile: "Solaris extended header" (no need to remove this one)

Solaris has many names and variants:

• SunOS
• SRV4
• Solaris
• Illumos (still active in July 2020 at least)
• OpenSolaris: "On Friday, August 13, 2010, details started to emerge relating to the discontinuation of the OpenSolaris project and the pending release of a new closed-source, proprietary version of Solaris, Solaris 11." says Wikipedia
• OpenIndiana (latest release in 2020)

I'm not sure on how to process? Deprecate all Solaris-specific features in Python 3.10, make sure that Python 3.11 cannot be built on Solaris anymore, and drop the code in Python 3.12? Or drop everything as soon as possible in Python 3.10?

The alternative is to leave the code unchanged and let the code slowly die, as we did for other platforms like HP-UX and IRIX.

See also my notes on platforms supported by Python:
https://pythondev.readthedocs.io/platforms.html

Attached draft PR gives an idea on how much code can be removed if Solaris support is removed. A coase estimation is the removal of 700 lines:

Doc/c-api/init.rst | 3 +-
Doc/distutils/apiref.rst | 1 -
Doc/library/gettext.rst | 10 +-
Doc/library/os.rst | 12 +-
Doc/library/platform.rst | 5 +-
Doc/library/posix.rst | 8 +-
Doc/library/sys.rst | 3 +-
Doc/library/sysconfig.rst | 1 -
Doc/library/time.rst | 13 +-
Doc/using/unix.rst | 7 -
Include/pyport.h | 5 -
Lib/cgi.py | 4 +-
Lib/ctypes/test/test_byteswap.py | 2 +-
Lib/ctypes/util.py | 100 ++--------
Lib/distutils/command/bdist.py | 2 +-
Lib/distutils/tests/support.py | 2 +-
Lib/distutils/tests/test_build_ext.py | 21 --
Lib/distutils/util.py | 11 --
Lib/gettext.py | 3 -
Lib/logging/handlers.py | 2 -
Lib/platform.py | 27 +--
Lib/sysconfig.py | 11 --
Lib/test/fork_wait.py | 4 +-
Lib/test/subprocessdata/fd_status.py | 3 -
Lib/test/test__locale.py | 24 ---
Lib/test/test_asyncio/test_sendfile.py | 6 -
Lib/test/test_asyncore.py | 2 -
Lib/test/test_cmd_line.py | 2 +-
Lib/test/test_curses.py | 2 +-
Lib/test/test_fileio.py | 2 +-
Lib/test/test_importlib/import_/test_path.py | 2 +-
Lib/test/test_locale.py | 2 +-
Lib/test/test_os.py | 14 +-
Lib/test/test_posix.py | 7 +-
Lib/test/test_pty.py | 4 -
Lib/test/test_selectors.py | 2 +-
Lib/test/test_shutil.py | 2 +-
Lib/test/test_socket.py | 4 -
Lib/test/test_strftime.py | 3 +-
Lib/test/test_sys.py | 2 +-
Lib/uuid.py | 5 -
Modules/Setup | 4 -
Modules/_ctypes/ctypes.h | 4 -
Modules/_cursesmodule.c | 2 +-
Modules/_multiprocessing/multiprocessing.h | 15 --
Modules/_posixsubprocess.c | 11 --
Modules/errnomodule.c | 23 +--
Modules/mathmodule.c | 2 +-
Modules/posixmodule.c | 65 +-----
Modules/signalmodule.c | 2 +-
Modules/socketmodule.c | 7 -
Modules/termios.c | 8 +-
Modules/timemodule.c | 9 +-
Python/bootstrap_hash.c | 17 +-
Python/fileutils.c | 3 +-
Python/import.c | 3 +-
Tools/freeze/README | 5 -
Tools/i18n/pygettext.py | 3 +-
Tools/pynche/Main.py | 2 -
Tools/pynche/README | 2 +-
Tools/scripts/nm2def.py | 2 +-
configure | 285 +--------------------------
configure.ac | 101 +---------
pyconfig.h.in | 19 +-
setup.py | 20 +-
65 files changed, 91 insertions(+), 868 deletions(-)

[wiedi]

Please don't.

Even if Oracle doesn't care anymore, the open source illumos community does.
We (illumos) share the same uname "SunOS" and depend on this support.

I have offered to host an illumos buildbot in the past and that offer is still valid.
At that time the testsuite did not pass and I could not figure out the fix yet.
If having this fixed is a requirement to keep illumos supported I will invest more time into this. I can also offer shell access to anyone interested in helping with this.

[vstinner]

I looked at PyPI download statistics of the "six" module over the last 180 days (6 months), grouped by operating system name:

• SunOS: 3,975 downloads
• TOTAL: 590,727,298 downloads

SunOS is 0.0007% of all six downloads.

For comparison, the top 4 is:

• Linux: 91% (535,804,817)
• Windows: 4% (21,053,173)
• <empty system name>: 3% (18,695,360)
• macOS (Darwin): 3% (14,776,993)

Raw data of the top 14:

num_downloads,system_name
535804817,Linux
21053173,Windows
18695360,
14776993,Darwin
347090,FreeBSD
9759,Qnx
5350,Java
3975,SunOS
3178,CYGWIN_NT-10.0-14393
3177,CYGWIN_NT-10.0-17763
2505,CYGWIN_NT-10.0-18363
2263,AIX
2061,NetBSD
1891,OpenBSD

[vstinner]

We (illumos) share the same uname "SunOS" and depend on this support.

Which Python version do you use?

https://blogs.oracle.com/solaris/future-of-python-on-solaris mentions Python 2.7 and 3.4.

[wiedi]

Which Python version do you use?

Myself currently mostly python 3.8 via pkgsrc on SmartOS (one of the illumos distributions).

OmniOS ships 3.7: https://github.com/omniosorg/omnios-build/tree/master/build/python37

OpenIndiana ships at least 2.7, 3.5, 3.7.

The illumos build process also depends on python and was updated to support 3.9 two weeks ago: https://www.illumos.org/issues/13217

[mimivx]

Oracle moved solaris 11.4 to +- rolling release mode, last SRU is 26 from 20.10.2020 with month release cadence.

On another side, almost all informations are behind the Oracle paywall

[jcea]

I think Solaris Support is still important and I could invest more time on it.

About package downloads:

1. I routinely download packages in my laptop (Linux) and distribute them to a fleet of servers (Linux and Solaris) via ansible and similar. Many servers in intranets don't install packages directly via PIP.

2. Under OmniOS or SmartOS (Illumos distributions), many people use the system package manager to install Python packages, not "PIP". For instance, http://netbsd.2816.n7.nabble.com/WIP-commit-Upgrade-to-Mailman-2-1-34-td427426.html.

Buildbots: I used to manage a buildbot with OpenIndiana, but it didn't survived the GITHUB migration. I could provide now a SmartOS buildbot.

As a core developer, I do care about Solaris. I fully realize I have been quite inactive since the github migration. My fault. I could devote more time to the task.

[jcea]

I am using Python 3.8 installed via pkgsrc in SmartOS with some Zones running Python 3.9 self compiled code.

More about the downloads stats:

4% packages in windows: I am pretty sure that most windows programmers install packages via conda-like tools. Surely there is more that 4% of python programmers on windows, even more in machine learning and data science areas, where windows and MacOS X rule (apparently being only 7% of PIP downloads).

Even if the PIP download stats would reflect penetration, Solaris is ahead of NetBSD and OpenBSD together.

[jclulow]

Hi! I'm a member of the illumos core team, and I'm also pretty keen for us to keep Python support! Some of our core OS build and packaging tooling is written in Python, and certainly applications like Synapse (Matrix) and Review Board are important to myself.

We currently provide a virtual machine for builds to the Go project that is wired in to their official CI/CD. I expect we can help out with a similar thing here!

I know some illumos distributions (OmniOS, OpenIndiana, anything with pkgsrc, etc) are maintaining some patches against the stock Python to fix failing tests, etc. Hopefully we can figure out how to get those things upstreamed and increase our participation there.

[citrus-it]

We at OmniOS (an illumos distribution) are in the process of upgrading to Python 3.9 and working on getting tests clean. We're in pretty good shape with a few local patches and only having to skip a few tests at the moment, and most of those patches should be suitable for integrating generally.

Some of the tests just need adjusting to be aware of the different platform (e.g. illumos does not support multiple SCM_RIGHTS messages in a single packet), some still need investigating to find the root cause, but the list is small.

[citrus-it]

Re: Downloads

OmniOS makes modules like 'six' available through its own IPS packaging system (which is written in Python) so installations won't show up pypi stats.

six is a core module on the platform so it is installed on 1000s of OmniOS machines (and then large numbers of SmartOS, OpenIndiana and tribblix ones too - all using the SunOS tag). The numbers aren't going to be in the same league as other platforms but the downloads stats definitely don't represent the illumos installation base and dependency on Python.