OpenSSL unsafe legacy renegotiation disabled error

[HASKADOG]

Description

I get an SSL issue on a working site
twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', '', 'unsafe legacy renegotiation disabled')]>]

Steps to Reproduce

1. scrapy shell https://dorotheum.com

Expected behavior: HTML page

Actual behavior: the error above

Reproduces how often: 100%

Versions

Scrapy       : 2.6.1
lxml         : 4.8.0.0
libxml2      : 2.9.4
cssselect    : 1.1.0
parsel       : 1.6.0
w3lib        : 1.22.0
Twisted      : 22.4.0
Python       : 3.9.12 (main, Mar 26 2022, 15:44:31) - [Clang 13.1.6 (clang-1316.0.21.2)]
pyOpenSSL    : 22.0.0 (OpenSSL 3.0.3 3 May 2022)
cryptography : 37.0.2
Platform     : macOS-12.2.1-arm64-arm-64bit

[wRAR]

This is explicitly disabled in OpenSSL 3: https://www.openssl.org/docs/manmaster/man7/migration_guide.html

We may need to add some new OP_* flags to support connecting to these outdated servers.

[HASKADOG]

@wRAR Can the downgrade of OpenSSL, pyopenssl ang cryprography help in this situation?

[wRAR]

You should downgrade pyOpenSSL to a version that ships 1.1, assuming it uses a bundled one in your environment (which I suspect it does).

[HASKADOG]

@wRAR looks like twisted does not support OP flasgs...

[wRAR]

You can usually access the underlying OpenSSL.SSL.Context object and call set_options() on it.

[HASKADOG]

@wRAR fixed by downgrading cryptography to 36.0.2
Here's my current scrapy version --verbose result

Scrapy       : 2.6.1
lxml         : 4.8.0.0
libxml2      : 2.9.4
cssselect    : 1.1.0
parsel       : 1.6.0
w3lib        : 1.22.0
Twisted      : 22.4.0
Python       : 3.9.12 (main, Mar 26 2022, 15:44:31) - [Clang 13.1.6 (clang-1316.0.21.2)]
pyOpenSSL    : 22.0.0 (OpenSSL 1.1.1n  15 Mar 2022)
cryptography : 36.0.2
Platform     : macOS-12.3.1-arm64-arm-64bit

Now pyOpenSSL uses OpenSSL 1.1.1n
Even though the problem is fixed, the flags enhancement would be great.
Thank you so much @wRAR !

[wRAR]

Right, it's cryptography that bundles libssl.

[Yoyoda75]

I'm having the same problem when trying to scrape the Cisco website for security advisories. How can I downgrade cryptography to a specific version inside of the scrapy package ?

[wRAR]

It's not "inside of the scrapy package", it's in your Python environment.

[dogweather]

I've having the same problem with Government of Ireland websites. E.g.,

scrapy fetch https://www.courts.ie/

Comes back with the error,

2022-07-05 17:33:05 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.courts.ie/robots.txt> (failed 1 times): [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', '', 'unsafe legacy renegotiation disabled')]>]

Curl and various web browsers do fine with it. Downgrading cryptography works. Thank you!

Does anyone know how to determine exactly what these websites need to upgrade? I.e., if I were going to get in touch with the Government of Ireland webmaster, what would I say? It's not immediately obvious that the bug is not with cryptography.

[wRAR]

If you are going to contact the website owner just point them to https://www.ssllabs.com/ssltest/ results for their website.

I mean, if the task is "make it work with OpenSSL 3.0 clients" then the fix is to implement RFC 5746, but I'm not sure the website owner would really want to fix it, considering that the website works in browsers, and it may not be easy or even possible with the stack they are now using, looking at its other problems. And if the task is "make it more secure" then the more important problem is SSL3 not being disabled.

It's not immediately obvious that the bug is not with cryptography.

I agree it's not immediately obvious and some research would be needed. I did it earlier and linked the OpenSSL 3.0 changelog which explicitly lists things that are not supported anymore.

[jeroenvermunt]

I want to drop this relevant link here:

https://stackoverflow.com/questions/71603314/ssl-error-unsafe-legacy-renegotiation-disabled/72374542#72374542

Here is a very useful piece of code to circumvent the issue without downgrading:

import requests
import urllib3
import SSL


class CustomHttpAdapter (requests.adapters.HTTPAdapter):
    # "Transport adapter" that allows us to use custom ssl_context.

    def __init__(self, ssl_context=None, **kwargs):
        self.ssl_context = ssl_context
        super().__init__(**kwargs)

    def init_poolmanager(self, connections, maxsize, block=False):
        self.poolmanager = urllib3.poolmanager.PoolManager(
            num_pools=connections, maxsize=maxsize,
            block=block, ssl_context=self.ssl_context)


def get_legacy_session():
    ctx = ssl.create_default_context(ssl.Purpose.SERVER_AUTH)
    ctx.options |= 0x4  # OP_LEGACY_SERVER_CONNECT
    session = requests.session()
    session.mount('https://', CustomHttpAdapter(ctx))
    return session

with (
    get_legacy_session() as s,
    s.get("some-url") as response
):
    print(response.json())

[wRAR]

(note that Scrapy doesn't use requests, so accessing the OpenSSL context from a spider is left as an exercise to the reader)

[aysegulc]

I ran into the same problem, and built a custom context factory to solve it. I hope it will be helpful for others:

contextfactory.py (within MyProject folder)

from scrapy.core.downloader.contextfactory import ScrapyClientContextFactory


class LegacyConnectContextFactory(ScrapyClientContextFactory):

    def getContext(self, hostname=None, port=None):
        ctx = self.getCertificateOptions().getContext()
        ctx.set_options(0x4)
        return ctx

within my_spider.py

class MySpider(Spider):
    name = "my_spider"

    custom_settings = {
        'DOWNLOADER_CLIENTCONTEXTFACTORY': 'MyProject.contextfactory.LegacyConnectContextFactory',
    }

[paradox-lab]

I ran into the same problem, and built a custom context factory to solve it. I hope it will be helpful for others:

contextfactory.py (within MyProject folder)

from scrapy.core.downloader.contextfactory import ScrapyClientContextFactory


class LegacyConnectContextFactory(ScrapyClientContextFactory):

    def getContext(self, hostname=None, port=None):
        ctx = self.getCertificateOptions().getContext()
        ctx.set_options(0x4)
        return ctx

this code works too.

contextfactory.py

from scrapy.core.downloader.contextfactory import ScrapyClientContextFactory
from cryptography.hazmat.bindings.openssl.binding import Binding


class LegacyConnectContextFactory(ScrapyClientContextFactory):

    def getContext(self, hostname=None, port=None):
        ctx = super(LegacyConnectContextFactory, self).getContext()
        binding = Binding()
        ctx.set_options(binding.lib.SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION)
        return ctx