Segfault 11 on Mac (10.10.5) with Python 3.4.4 for outlook.com's IMAP

[mjs]

Originally reported by: Paul Hammant (Bitbucket: paul_hammant)

---

$ python3 testtt.py
Traceback (most recent call last):
File "testtt.py", line 18, in
print('%d messages in INBOX' % select_info['EXISTS'])
KeyError: 'EXISTS'
Segmentation fault: 11

1. I've done DiskUtility - disk repair (sometimes people ask you to do that for segfault-y bug reports)
2. I uninstalled Python3.5.1 from homebrew, and did the downloadable Python 3.4.4 installer from Python.org.
3. Using it's pip3 command, I installed imapclient today.
4. I took your first script from the https://imapclient.readthedocs.org site, and changed imap server, user, password (and turned on SSL).
5. Running you script causes the seg-fault, as above. The Apple app-crash-catcher opens up. I've dismissed it. This is consistently reproducible for me, so I can run it again, and send debug information.

I can reproduce this on two Macs. One a MacBookAir (2011), one a MacBookPro (2014). Both running Yosemite 10.10.5. I was trying first with Python 3.5.1, but I note that is not listed as supported, so I downgraded and tried again.

I would test the same script on Windows10/Python334 but I get SSL errors. See below.

Note - I was logging issues with backport.ssl too - alekstorm/backports.ssl#16 and alekstorm/backports.ssl#15 and alekstorm/backports.ssl#14

---

• Bitbucket: https://bitbucket.org/mjs0/imapclient/issue/201

[mjs]

Original comment by Menno Smits (Bitbucket: mjs0):

---

The only way you could be getting a segfault from IMAPClient is if one of the C extensions it depends on is failing. The likely candidates are pyOpenSSL, cryptography.io and cffi.

Possibly related tickets:

• https://bitbucket.org/cffi/cffi/issues/246/crash-during-interpreter-shutdown-when
• Segfault on python3.5.1, cryptography 1.2.1 but not on 1.1.2 [MAC-EL CAPITAN] pyca/cryptography#2690

Is this problem still happening for you? If so could you please run the following from a Python shell in your environment and post the output here?

import cffi, cryptography, OpenSSL

print("cffi: ", cffi.__version__)
print("cryptography: ", cryptography.__version__)
print("pyOpenSSL: ", OpenSSL.__version__)

[mjs]

Original comment by Paul Hammant (Bitbucket: paul_hammant):

---

Output from your script, Menno, after pip3 install all three today:

#!bash
cffi:  1.6.0
cryptography:  1.4
pyOpenSSL:  16.0.0

I've new work Mac, so I'm stating over on this. The script I refer to from the https://imapclient.readthedocs.org page:

#!python

$ python3 foo.py 
140 messages in INBOX
140 messages that aren't deleted

Messages:
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/backports/ssl/core.py", line 201, in _safe_ssl_call
    return getattr(sock, call)(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/OpenSSL/SSL.py", line 1271, in send
    self._raise_ssl_error(self._ssl, result)
  File "/usr/local/lib/python3.5/site-packages/OpenSSL/SSL.py", line 1182, in _raise_ssl_error
    raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (32, 'EPIPE')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/imaplib.py", line 963, in _command
    self.send(data + CRLF)
  File "/usr/local/lib/python3.5/site-packages/imapclient/tls.py", line 171, in send
    sent = self.sock.send(data)
  File "/usr/local/lib/python3.5/site-packages/backports/ssl/core.py", line 281, in send
    return _safe_ssl_call(False, self._conn, 'send', data, flags)
  File "/usr/local/lib/python3.5/site-packages/backports/ssl/core.py", line 216, in _safe_ssl_call
    raise SSLSysCallError(*e.args)
backports.ssl.core.SSLSysCallError: [Errno 32] EPIPE

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "foo.py", line 32, in <module>
    response = server.fetch(messages, ['FLAGS', 'RFC822.SIZE'])
  File "/usr/local/lib/python3.5/site-packages/imapclient/imapclient.py", line 971, in fetch
    tag = self._imap._command(*args)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/imaplib.py", line 965, in _command
    raise self.abort('socket error: %s' % val)
imaplib.abort: socket error: [Errno 32] EPIPE
HSE-NYC-MAC-03:hs phammant$ 

Sure, that's a different error. Other Python projects are reporting "[Errno 32] EPIPE".

Homebrew put Python 3.5.1 on the Mac. Should I roll back to 3.4 and retest ?

[mjs]

Original comment by Paul Hammant (Bitbucket: paul_hammant):

---

Incidentally, I think the dict lookup should be b'EXISTS' not 'EXISTS'

[mjs]

Original comment by Menno Smits (Bitbucket: mjs0):

---

If it's not too hard for you to try with Python 3.4, then yes please.

Which version of OS X is the new laptop running?

This is certainly a different problem to what you were seeing before, given that this is an exception and you were seeing a segfault before. Based on what I've been reading it's possible the segfault was fixed in cffi 0.15.1 so I wonder if you were running an older version of cffi earlier and are now past that?

There's been a bunch of problems people running pyOpenSSL on OS X. Very frustrating. I'm investigating the possibility of making the use of backports.ssl and pyOpenSSL etc optional, especially for versions of Python with improved SSL support built-in (2.7.9 and later, and 3.4 and later).

[mjs]

Original comment by Paul Hammant (Bitbucket: paul_hammant):

---

Installing with Python 3.4 (tricky with homebrew - https://github.com/zoidbergwill/homebrew-python)

#!python
Installing collected packages: pyasn1, pycparser, cffi, idna, six, cryptography, pyopenssl, pbr, mock, backports.ssl, imapclient
Successfully installed backports.ssl-0.0.9 cffi-1.6.0 cryptography-1.4 idna-2.1 imapclient-1.0.1 mock-1.3.0 pbr-1.10.0 pyasn1-0.1.9 pycparser-2.14 pyopenssl-16.0.0 six-1.10.0

Trying the https://imapclient.readthedocs.org example (with the b'EXISTS' fix):

#!python
$ python3.4 foo.py 
Traceback (most recent call last):
  File "foo.py", line 21, in <module>
    server.login(USERNAME, PASSWORD)
  File "/usr/local/lib/python3.4/site-packages/imapclient/imapclient.py", line 215, in login
    unpack=True,
  File "/usr/local/lib/python3.4/site-packages/imapclient/imapclient.py", line 1180, in _command_and_check
    typ, data = meth(*args)
  File "/usr/local/Cellar/python34/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/imaplib.py", line 538, in login
    raise self.error(dat[-1])
imaplib.error: b'LOGIN failed.'
HSE-NYC-MAC-03:hs phammant$ atom foo.py 
HSE-NYC-MAC-03:hs phammant$ python3.4 foo.py 
dict_keys([b'RECENT', b'READ-WRITE', b'FLAGS', b'PERMANENTFLAGS', b'EXISTS', b'UNSEEN', b'UIDNEXT', b'UIDVALIDITY'])
54272 messages in INBOX
54272 messages that aren't deleted

Messages:
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/site-packages/backports/ssl/core.py", line 201, in _safe_ssl_call
    return getattr(sock, call)(*args, **kwargs)
  File "/usr/local/lib/python3.4/site-packages/OpenSSL/SSL.py", line 1271, in send
    self._raise_ssl_error(self._ssl, result)
  File "/usr/local/lib/python3.4/site-packages/OpenSSL/SSL.py", line 1182, in _raise_ssl_error
    raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (32, 'EPIPE')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/python34/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/imaplib.py", line 916, in _command
    self.send(data + CRLF)
  File "/usr/local/lib/python3.4/site-packages/imapclient/tls.py", line 171, in send
    sent = self.sock.send(data)
  File "/usr/local/lib/python3.4/site-packages/backports/ssl/core.py", line 281, in send
    return _safe_ssl_call(False, self._conn, 'send', data, flags)
  File "/usr/local/lib/python3.4/site-packages/backports/ssl/core.py", line 216, in _safe_ssl_call
    raise SSLSysCallError(*e.args)
backports.ssl.core.SSLSysCallError: [Errno 32] EPIPE

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "foo.py", line 32, in <module>
    response = server.fetch(messages, ['FLAGS', 'RFC822.SIZE'])
  File "/usr/local/lib/python3.4/site-packages/imapclient/imapclient.py", line 971, in fetch
    tag = self._imap._command(*args)
  File "/usr/local/Cellar/python34/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/imaplib.py", line 918, in _command
    raise self.abort('socket error: %s' % val)
imaplib.abort: socket error: [Errno 32] EPIPE

OSX is 10.11.5

If you could leave, backports.ssl (etc) behind, I think you'd be better off.
My beloved Pi0 has Python 2.7.9 (Raspbian), now :)

[NicolasLM]

Probably related to an incompatibility between MacOS/Python/backports.ssl. I believe this issue can be closed.

[mjs]

IMAPClient 2.x will no longer use backports.ssl etc. This problem will go away when that happens.