Use of TLS_METHOD forces requirement of pyOpenSSL >= 21.0

[cas--]

A Deluge user running Debian 10 on Raspberry pi 4 reported the following failure:

  File "/usr/local/lib/python3.7/dist-packages/twisted/internet/_sslverify.py", line 1829, in <module>
    "TLS13-AES-256-GCM-SHA384:TLS13-CHACHA20-POLY1305-SHA256:"
  File "/usr/local/lib/python3.7/dist-packages/twisted/internet/_sslverify.py", line 1807, in fromOpenSSLCipherString
    SSL.TLS_METHOD,
AttributeError: module 'OpenSSL.SSL' has no attribute 'TLS_METHOD'

A bit of searching also found a FreeBSD issue: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=268043

The issue stems from replacing deprecated TLS version references but TLS_METHOD was only added to pyOpenSSL 21.0.0 in
pyca/pyopenssl@5dc6988 released Sep '21 last year.

Would a fallback be considered to accommodate those on older versions of pyOpenSSL to TLSv1_2_METHOD?

In the meantime I will suggest users to either upgrade pyOpenSSL >= 21 or downgrade Twisted <= 22.4

[thisAmericanLife]

I'm impacted by this as well.

[adiroiban]

Hi,

Thanks for the report.

I am not sure what is the cause of this issue here.

From what I can see, for the 22.10.0 release, Twisted requires pyopenssl >= 21.0.0.

Is this not enough?

I think a fallback can be accepted, but someone will need to work on that fallback.

Cheers

[cas--]

It is requirement but since it is an extra it is likely is not (and cannot) enforced. I imagine these issues come from packaging from source without using pip.

I know it's a tricky line to tread with dependencies and security but I am concerned with the hard requirement for such a important dependency, with minimum version not even a year old at the time of merge, when the previous minimum version was from 2016.

[exarkun]

If the problem is that folks are upgrading to a new version of Twisted incorrectly then -- apart from suggesting that they install it correctly -- is there something wrong with suggesting that they also upgrade to a new version of pyOpenSSL?

That is, is there some situation in which it is possible to upgrade Twisted and not pyOpenSSL?

[thisAmericanLife]

In the meantime I will suggest users to either upgrade pyOpenSSL >= 21 or downgrade Twisted <= 22.4

Upgrading to pyOpenSSL 22 did not resolve this error for me.

Downgrading Twisted to 22.4 did.

[exarkun]

In the meantime I will suggest users to either upgrade pyOpenSSL >= 21 or downgrade Twisted <= 22.4

Upgrading to pyOpenSSL 22 did not resolve this error for me.

I don't know why this would be. pyOpenSSL 22 unconditionally defines OpenSSL.SSL.TLS_METHOD:

https://github.com/pyca/pyopenssl/blob/22.0.0/src/OpenSSL/SSL.py#L145

Are you sure you upgraded pyOpenSSL successfully and in the right Python environment? If so, can you share instructions for reproducing this result?

[thisAmericanLife]

Are you sure you upgraded pyOpenSSL successfully and in the right Python environment? If so, can you share instructions for reproducing this result?

I'm using twisted/klein.

In my environment I have pyOpenSSL 22, and I got the 'OpenSSL.SSL' has no attribute 'TLS_METHOD' before downgrading twisted when I attempted to import klein (i.e.: from klein import Klein).

[exarkun]

Are you sure you upgraded pyOpenSSL successfully and in the right Python environment? If so, can you share instructions for reproducing this result?

I'm using twisted/klein.

In my environment I have pyOpenSSL 22, and I got the 'OpenSSL.SSL' has no attribute 'TLS_METHOD' before downgrading twisted when I attempted to import klein (i.e.: from klein import Klein).

Thanks - however, these aren't instructions for reproducing your results. How did you get "your environment"? Here's an example:

❯ , python3.10 -m venv /tmp/pyopenssl
❯ . /tmp/pyopenssl/bin/activate
❯ pip install twisted[tls]
Collecting twisted[tls]
  Using cached Twisted-22.10.0-py3-none-any.whl (3.1 MB)
Collecting zope.interface>=4.4.2
  Using cached zope.interface-5.5.2-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (258 kB)
Collecting attrs>=19.2.0
  Using cached attrs-22.1.0-py2.py3-none-any.whl (58 kB)
Collecting typing-extensions>=3.6.5
  Using cached typing_extensions-4.4.0-py3-none-any.whl (26 kB)
Collecting incremental>=21.3.0
  Using cached incremental-22.10.0-py2.py3-none-any.whl (16 kB)
Collecting constantly>=15.1
  Using cached constantly-15.1.0-py2.py3-none-any.whl (7.9 kB)
Collecting Automat>=0.8.0
  Using cached Automat-22.10.0-py2.py3-none-any.whl (26 kB)
Collecting hyperlink>=17.1.1
  Using cached hyperlink-21.0.0-py2.py3-none-any.whl (74 kB)
Collecting pyopenssl>=21.0.0
  Using cached pyOpenSSL-22.1.0-py3-none-any.whl (57 kB)
Collecting service-identity>=18.1.0
  Using cached service_identity-21.1.0-py2.py3-none-any.whl (12 kB)
Collecting idna>=2.4
  Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting six
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting cryptography<39,>=38.0.0
  Using cached cryptography-38.0.4-cp36-abi3-manylinux_2_28_x86_64.whl (4.2 MB)
Collecting pyasn1-modules
  Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting pyasn1
  Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Requirement already satisfied: setuptools in /tmp/pyopenssl/lib/python3.10/site-packages (from zope.interface>=4.4.2->twisted[tls]) (58.1.0)
Collecting cffi>=1.12
  Using cached cffi-1.15.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (441 kB)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Installing collected packages: pyasn1, incremental, constantly, zope.interface, typing-extensions, six, pycparser, pyasn1-modules, idna, attrs, hyperlink, cffi, Automat, twisted, cryptography, service-identity, pyopenssl
Successfully installed Automat-22.10.0 attrs-22.1.0 cffi-1.15.1 constantly-15.1.0 cryptography-38.0.4 hyperlink-21.0.0 idna-3.4 incremental-22.10.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.21 pyopenssl-22.1.0 service-identity-21.1.0 six-1.16.0 twisted-22.10.0 typing-extensions-4.4.0 zope.interface-5.5.2
WARNING: You are using pip version 22.0.4; however, version 22.3.1 is available.
You should consider upgrading via the '/tmp/pyopenssl/bin/python3.10 -m pip install --upgrade pip' command.

❯ python -c 'from OpenSSL import SSL; print(SSL.TLS_METHOD)'
7
❯ pip install klein
Collecting klein
  Using cached klein-21.8.0-py2.py3-none-any.whl (88 kB)
Collecting Werkzeug
  Using cached Werkzeug-2.2.2-py3-none-any.whl (232 kB)
Requirement already satisfied: Twisted>=16.6 in /tmp/pyopenssl/lib/python3.10/site-packages (from klein) (22.10.0)
Collecting Tubes
  Using cached Tubes-0.2.1-py2.py3-none-any.whl (59 kB)
Requirement already satisfied: zope.interface in /tmp/pyopenssl/lib/python3.10/site-packages (from klein) (5.5.2)
Requirement already satisfied: attrs in /tmp/pyopenssl/lib/python3.10/site-packages (from klein) (22.1.0)
Requirement already satisfied: hyperlink in /tmp/pyopenssl/lib/python3.10/site-packages (from klein) (21.0.0)
Requirement already satisfied: incremental in /tmp/pyopenssl/lib/python3.10/site-packages (from klein) (22.10.0)
Requirement already satisfied: typing-extensions>=3.6.5 in /tmp/pyopenssl/lib/python3.10/site-packages (from Twisted>=16.6->klein) (4.4.0)
Requirement already satisfied: constantly>=15.1 in /tmp/pyopenssl/lib/python3.10/site-packages (from Twisted>=16.6->klein) (15.1.0)
Requirement already satisfied: Automat>=0.8.0 in /tmp/pyopenssl/lib/python3.10/site-packages (from Twisted>=16.6->klein) (22.10.0)
Requirement already satisfied: idna>=2.5 in /tmp/pyopenssl/lib/python3.10/site-packages (from hyperlink->klein) (3.4)
Requirement already satisfied: setuptools in /tmp/pyopenssl/lib/python3.10/site-packages (from zope.interface->klein) (58.1.0)
Requirement already satisfied: six in /tmp/pyopenssl/lib/python3.10/site-packages (from Tubes->klein) (1.16.0)
Collecting MarkupSafe>=2.1.1
  Using cached MarkupSafe-2.1.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Installing collected packages: MarkupSafe, Werkzeug, Tubes, klein
Successfully installed MarkupSafe-2.1.1 Tubes-0.2.1 Werkzeug-2.2.2 klein-21.8.0
WARNING: You are using pip version 22.0.4; however, version 22.3.1 is available.
You should consider upgrading via the '/tmp/pyopenssl/bin/python3.10 -m pip install --upgrade pip' command.

❯ python -c 'from klein import Klein; print(Klein)'
<class 'klein._app.Klein'>

[karolyi]

The same thing happens on FreeBSD 13.1 currently with synapse (matrix homeserver) install.

It pulls py39-openssl-20.0.1,1 and py39-twisted-22.10.0, and then you get this error.

I'll notify the maintainer as well.

[karolyi]

Related: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=268043

[charly37]

Same issue today on new install of a PI with:

Raspberry Pi OS (Legacy) with desktop
Release date: December 5th 2023
System: 32-bit
Kernel version: 6.1
Debian version: 11 (bullseye)

Same error when trying http example:

charles@raspberrypi:~ $ python main.py
Traceback (most recent call last):
  File "/home/charles/main.py", line 2, in <module>
    from twisted.internet import reactor, endpoints
  File "/usr/local/lib/python3.9/dist-packages/twisted/internet/reactor.py", line 38, in <module>
    from twisted.internet import default
  File "/usr/local/lib/python3.9/dist-packages/twisted/internet/default.py", line 55, in <module>
    install = _getInstallFunction(platform)
  File "/usr/local/lib/python3.9/dist-packages/twisted/internet/default.py", line 43, in _getInstallFunction
    from twisted.internet.epollreactor import install
  File "/usr/local/lib/python3.9/dist-packages/twisted/internet/epollreactor.py", line 19, in <module>
    from twisted.internet import posixbase
  File "/usr/local/lib/python3.9/dist-packages/twisted/internet/posixbase.py", line 16, in <module>
    from twisted.internet import error, tcp, udp
  File "/usr/local/lib/python3.9/dist-packages/twisted/internet/tcp.py", line 38, in <module>
    from twisted.internet._newtls import (
  File "/usr/local/lib/python3.9/dist-packages/twisted/internet/_newtls.py", line 18, in <module>
    from twisted.protocols.tls import TLSMemoryBIOFactory
  File "/usr/local/lib/python3.9/dist-packages/twisted/protocols/tls.py", line 48, in <module>
    from twisted.internet._sslverify import _setAcceptableProtocols
  File "/usr/local/lib/python3.9/dist-packages/twisted/internet/_sslverify.py", line 1826, in <module>
    defaultCiphers = OpenSSLAcceptableCiphers.fromOpenSSLCipherString(
  File "/usr/local/lib/python3.9/dist-packages/twisted/internet/_sslverify.py", line 1805, in fromOpenSSLCipherString
    SSL.TLS_METHOD,
AttributeError: module 'OpenSSL.SSL' has no attribute 'TLS_METHOD'

Works fine after forcing an upgrade of pyopenssl:

charles@raspberrypi:~ $ sudo pip install pyOpenSSL -U
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Requirement already satisfied: pyOpenSSL in /usr/lib/python3/dist-packages (20.0.1)
Collecting pyOpenSSL
  Downloading https://www.piwheels.org/simple/pyopenssl/pyOpenSSL-23.3.0-py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 190 kB/s
Collecting cryptography<42,>=41.0.5
  Downloading https://www.piwheels.org/simple/cryptography/cryptography-41.0.7-cp39-cp39-linux_armv7l.whl (2.1 MB)
     |████████████████████████████████| 2.1 MB 298 kB/s
Collecting cffi>=1.12
  Downloading https://www.piwheels.org/simple/cffi/cffi-1.16.0-cp39-cp39-linux_armv7l.whl (367 kB)
     |████████████████████████████████| 367 kB 248 kB/s
Collecting pycparser
  Downloading https://www.piwheels.org/simple/pycparser/pycparser-2.21-py2.py3-none-any.whl (119 kB)
     |████████████████████████████████| 119 kB 348 kB/s
Installing collected packages: pycparser, cffi, cryptography, pyOpenSSL
  Attempting uninstall: cryptography
    Found existing installation: cryptography 3.3.2
    Not uninstalling cryptography at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'cryptography'. No files were found to uninstall.
  Attempting uninstall: pyOpenSSL
    Found existing installation: pyOpenSSL 20.0.1
    Not uninstalling pyopenssl at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'pyOpenSSL'. No files were found to uninstall.
Successfully installed cffi-1.16.0 cryptography-41.0.7 pyOpenSSL-23.3.0 pycparser-2.21

then no more error.... hoping it help other people. Also, i think on a different Pi it may work but since i use a Pi Zero I'm force to use "Pi OS (Legacy)"

[glyph]

I don't think that we are going to support downgrading security-critical dependencies such as pyOpenSSL to versions that are multiple years old. This seems like a problem for raspberry pi's maintainers to sort out.

If there's something we can do to make this easier, I'd be happy to explore it, but as stated (support older versions of pyOpenSSL) this is not something Twisted is going to do.