Fix SMTP STARTTLS for Twisted >= 21.2.0

[TobiMayr]

Summary

The Mail settings don't have an option to choose a TLS version. Only to enforce upgrading connections to use SSL/TLS.
Mail servers like smtp.office365.com dropped support for TLS1.0 and TLS1.1 and now require TLS1.2: https://techcommunity.microsoft.com/t5/exchange-team-blog/new-opt-in-endpoint-available-for-smtp-auth-clients-still/ba-p/2659652

It seems that scrapy mail doesn't support TLS1.2. The error message (with MAIL_TLS = True):

[scrapy.mail] Unable to send mail: To=['user@gmail.com'] Cc=[] Subject="Test" Attachs=0- 421 b'4.7.66 TLS 1.0 and 1.1 are not supported. Please upgrade/update your client to support TLS 1.2. Visit https://aka.ms/smtp_auth_tls. [AM6P194CA0047.EURP194.PROD.OUTLOOK.COM]'

Motivation

Without TLS1.2 it's not possible anymore to send mails via smtp.office365.com. An option to use TLS1.2 would fix this issue

[wRAR]

Scrapy doesn't configure protocol versions explicitly when using ESMTPSenderFactory and neither does that class, so I would expect it to use global OpenSSL settings, including TLS 1.2 or 1.3 if supported. See https://twistedmatrix.com/documents/current/core/howto/ssl.html

Can you show the library versions printed by Scrapy?

[TobiMayr]

It depends on the version of scrapy and which library version it uses for OpenSSL?
I use scrapy 2.0.1.
Scrapy prints the following when starting a job:
[scrapy.utils.log] Versions: lxml 4.5.0.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 19.10.0, Python 3.8.2 (default, Feb 26 2020, 15:09:34) - [GCC 8.3.0], pyOpenSSL 19.1.0 (OpenSSL 1.1.1d 10 Sep 2019), cryptography 2.8, Platform Linux-4.15.0-66-generic-x86_64-with-glibc2.2.5
Thanks for your help!

[wRAR]

I suggest trying newer pyOpenSSL/cryptography/Twisted versions first.

[TobiMayr]

I upgraded the versions to the newest:
cryptography==36.0.1 pyOpenSSL==22.0.0 twisted==21.7.0

Now I get a new error:
502 Server does not support secure communication via TLS / SSL

So it definitely looks like the issue is my mail server now. I will close this feature request.
Thank you for your quick help

[TobiMayr]

Hi @wRAR I might still need your help.

The smtp server supports TLS1.2. I also tested this by creating a script with smtplib.
This script successfully sends my email using TLS1.2:

import smtplib, ssl

port = 587
smtp_server = 'smtp.office365.com'
sender_email = 'test1@test.com'
receiver_email = 'test2@test.com'
user = 'test3@test.de'
password = input('Type your password and press enter:')
message = 'test'

context = ssl.create_default_context()
context.minimum_version = ssl.TLSVersion.TLSv1_2
with smtplib.SMTP(smtp_server, port) as server:
    server.ehlo()  # Can be omitted
    server.starttls(context=context)
    server.ehlo()  # Can be omitted
    server.login(user, password)
    server.sendmail(sender_email, receiver_email, message)

I can't seem to get it to work with scrapy. I created a new scrapy project with a newer version(2.5.1) as well.
These are all my versions:
[scrapy.utils.log] INFO: Versions: lxml 4.7.1.0, libxml2 2.9.12, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 21.7.0, Python 3.9.7 (default, Sep 3 2021, 12:37:55) - [Clang 12.0.5 (clang-1205.0.22.9)], pyOpenSSL 22.0.0 (OpenSSL 1.1.1m 14 Dec 2021), cryptography 36.0.1, Platform macOS-11.6.2-x86_64-i386-64bit

I still get the error:
[scrapy.mail] ERROR: Unable to send mail: To=['test@test.com'] Cc=[] Subject="tls-test" Attachs=0- 502 Server does not support secure communication via TLS / SSL

Do you have an idea what I could try next?

[wRAR]

Are you using MAIL_TLS and port 587?

[TobiMayr]

Both yes

[wRAR]

I'll try to check this later

[TobiMayr]

Hi @wRAR, sorry to bother you.
Do you have some more ideas of what I could try?