ImportError: No module named 'twisted.mail.smtp'

[duangy]

I use scrapy in python3.5, when I try to send a mail by MailSender, it's crashed.

  File "/home/guangyao/Documents/Code/financer_crawl/env/lib/python3.5/site-packages/scrapy/mail.py", line 85, in send
    dfd = self._sendmail(rcpts, msg.as_string())
  File "/home/guangyao/Documents/Code/financer_crawl/env/lib/python3.5/site-packages/scrapy/mail.py", line 108, in _sendmail
    from twisted.mail.smtp import ESMTPSenderFactory
ImportError: No module named 'twisted.mail.smtp'

[redapple]

@duangy , thanks for reporting.
Unfortunately, Scrapy does not support sending mail with Python 3.
This is actually a limitation of Twisted, which has not ported its twisted.mail module yet:
see https://twistedmatrix.com/trac/ticket/8770 and twisted/twisted#509 (comment)

[MrLokans]

I confirm having the same issue on the latest scrapy on python 3.5. @redapple , so, before twisted will fix that issue, what is the proper way of sending email reports from scrapy on python 3+ installations? Thank you.

[kmike]

@MrLokans I think you can use smtplib and email packages from standard library; see https://docs.python.org/3/library/email-examples.html. They are blocking though - it means that all processing will stop while email is waiting to be sent. So make sure sending is fast: run a local daemon (exim, postfix) and use it to send emails, so that when you call "send email" the message is put to a local queue, and then daemon handles the rest. Using e.g. gmail smtp interface directly is worse because it can be much slower.

[MrLokans]

@kmike Thanks a lot for the reply.

[TilakMaddy]

hmm .. learnt something today

[rodrigc]

@redapple I've gone ahead and merged twisted/twisted#509 .
It is not in any released version of Twisted yet.
If you are willing to try out Twisted directly from git, can you try it and and let me know how it goes?

[yalopov]

Someone knows if this issue is fixed? i'm using python 3.5, twisted 17.1.0, scrapy 1.3.0 and still can't use email service.

[rodrigc]

I submitted #2671
which needs to be merged.

[redapple]

@rodrigc , I tested your patch with (today's) Twisted trunk (commit 0aef33642cbe04cac558a4aaceeb2a6a03de403b),

and with Python 3.5 I needed to patch it further like this to make scrapy work for sending emails (for example sending crawl stats with scrapy.extensions.statsmailer.StatsMailer):

$ git diff
diff --git a/src/twisted/mail/smtp.py b/src/twisted/mail/smtp.py
index 8d0385d..f53dd06 100644
--- a/src/twisted/mail/smtp.py
+++ b/src/twisted/mail/smtp.py
@@ -990,7 +990,6 @@ class SMTPClient(basic.LineReceiver, policies.TimeoutMixin):
         if line[3:4] == b'-':
             # Continuation
             return
-
         if self.code in self._expected:
             why = self._okresponse(self.code, b'\n'.join(self.resp))
         else:
@@ -1569,7 +1568,7 @@ class ESMTPClient(SMTPClient):
             resp = auth.challengeResponse(self.secret, challenge)
             self._expected = [235, 334]
             self._okresponse = self.smtpState_maybeAuthenticated
-            self.sendLine(base64.b64encode(resp))
+            self.sendLine(base64.b64encode(networkString(resp)))
 
 
     def smtpState_maybeAuthenticated(self, code, resp):
@@ -1885,7 +1884,6 @@ class SMTPSenderFactory(protocol.ClientFactory):
         server responses, or None to wait forever.
         """
         assert isinstance(retries, (int, long))
-
         if isinstance(toEmail, unicode):
             toEmail = [toEmail.encode('ascii')]
         elif isinstance(toEmail, bytes):
@@ -1896,9 +1894,8 @@ class SMTPSenderFactory(protocol.ClientFactory):
                 if not isinstance(_email, bytes):
                     _email = _email.encode('ascii')
 
-                toEmailFinal.append(email)
+                toEmailFinal.append(_email)
             toEmail = toEmailFinal
-
         self.fromEmail = Address(fromEmail)
         self.nEmails = len(toEmail)
         self.toEmail = toEmail

The toEmailFinal.append(_email) line looks like a bug indeed, and is also an issue with Python 2.7.

The other one about converting the challenge response to bytes before base64 encoding it, I'm not sure it's the best place for it. I'll let you find a proper way.

(Otherwise, without that networkString call, I get this:

Traceback (most recent call last):
  File "/home/paul/src/twisted/src/twisted/python/log.py", line 103, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/home/paul/src/twisted/src/twisted/python/log.py", line 86, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/home/paul/src/twisted/src/twisted/python/context.py", line 122, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/home/paul/src/twisted/src/twisted/python/context.py", line 85, in callWithContext
    return func(*args,**kw)
--- <exception caught here> ---
  File "/home/paul/src/twisted/src/twisted/internet/posixbase.py", line 597, in _doReadOrWrite
    why = selectable.doRead()
  File "/home/paul/src/twisted/src/twisted/internet/tcp.py", line 208, in doRead
    return self._dataReceived(data)
  File "/home/paul/src/twisted/src/twisted/internet/tcp.py", line 214, in _dataReceived
    rval = self.protocol.dataReceived(data)
  File "/home/paul/src/twisted/src/twisted/protocols/tls.py", line 330, in dataReceived
    self._flushReceiveBIO()
  File "/home/paul/src/twisted/src/twisted/protocols/tls.py", line 295, in _flushReceiveBIO
    ProtocolWrapper.dataReceived(self, bytes)
  File "/home/paul/src/twisted/src/twisted/protocols/policies.py", line 120, in dataReceived
    self.wrappedProtocol.dataReceived(data)
  File "/home/paul/src/twisted/src/twisted/protocols/basic.py", line 571, in dataReceived
    why = self.lineReceived(line)
  File "/home/paul/src/twisted/src/twisted/mail/smtp.py", line 994, in lineReceived
    why = self._okresponse(self.code, b'\n'.join(self.resp))
  File "/home/paul/src/twisted/src/twisted/mail/smtp.py", line 1555, in esmtpState_challenge
    self._authResponse(self._authinfo, resp)
  File "/home/paul/src/twisted/src/twisted/mail/smtp.py", line 1572, in _authResponse
    self.sendLine(base64.b64encode(resp))
  File "/home/paul/.virtualenvs/scrapydev-twistedtrunk.py3/lib/python3.5/base64.py", line 59, in b64encode
    encoded = binascii.b2a_base64(s)[:-1]
builtins.TypeError: a bytes-like object is required, not 'str'

with SMTPSenderFactory.__init__(fromEmail='user@example.com', toEmail=['anotheruser@example.com']... and port 587)

[RobinDavid]

Hi @redapple & @rodrigc !
Nice merge request for Twisted! It does not seems to have been integrated in the latest version of Twisted 17.9.0. Do you know when will it be ? You mentioned some changes in Scrapy also, are they available in the 1.4.0 ? (it looks like they are still in trunk)

[redapple]

Hey @RobinDavid , with Twisted 17.9, I think #2671 is sufficient (but it isn't merged)