Test suite crashes on OS X w/ SIGBUS

[hynek]

This is a new one:

* thread #1: tid = 0x15138, 0x0000000104c73c40 _openssl.so`EC_GFp_nistp224_method, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x104c73c40)
    frame #0: 0x0000000104c73c40 _openssl.so`EC_GFp_nistp224_method
_openssl.so`EC_GFp_nistp224_method:
->  0x104c73c40 <+0>: addb   %al, (%rax)
    0x104c73c42 <+2>: addb   %al, (%rax)
    0x104c73c44 <+4>: addb   %al, (%rax)
    0x104c73c46 <+6>: addb   %al, (%rax)

The backtrace:

* thread #1: tid = 0x15138, 0x0000000104c73c40 _openssl.so`EC_GFp_nistp224_method, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x104c73c40)
  * frame #0: 0x0000000104c73c40 _openssl.so`EC_GFp_nistp224_method
    frame #1: 0x0000000104a6b749 _openssl.so`EC_GROUP_new_by_curve_name + 249
    frame #2: 0x0000000104a6e2d1 _openssl.so`EC_KEY_new_by_curve_name + 37
    frame #3: 0x0000000104b37fbd _openssl.so`_cffi_f_EC_KEY_new_by_curve_name + 61
    frame #4: 0x00000001000ac583 python`PyEval_EvalFrameEx + 16963
    frame #5: 0x00000001000af946 python`fast_function + 342
    frame #6: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #7: 0x00000001000af946 python`fast_function + 342
    frame #8: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #9: 0x00000001000af946 python`fast_function + 342
    frame #10: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #11: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #12: 0x00000001000af866 python`fast_function + 118
    frame #13: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #14: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #15: 0x000000010003162c python`function_call + 364
    frame #16: 0x000000010000b853 python`PyObject_Call + 99
    frame #17: 0x00000001000ab65c python`PyEval_EvalFrameEx + 13084
    frame #18: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #19: 0x000000010003162c python`function_call + 364
    frame #20: 0x000000010000b853 python`PyObject_Call + 99
    frame #21: 0x0000000100018af7 python`instancemethod_call + 247
    frame #22: 0x000000010000b853 python`PyObject_Call + 99
    frame #23: 0x00000001000687d5 python`slot_tp_call + 133
    frame #24: 0x000000010000b853 python`PyObject_Call + 99
    frame #25: 0x00000001000acbc7 python`PyEval_EvalFrameEx + 18567
    frame #26: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #27: 0x00000001000af866 python`fast_function + 118
    frame #28: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #29: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #30: 0x000000010003162c python`function_call + 364
    frame #31: 0x000000010000b853 python`PyObject_Call + 99
    frame #32: 0x00000001000ab65c python`PyEval_EvalFrameEx + 13084
    frame #33: 0x00000001000af946 python`fast_function + 342
    frame #34: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #35: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #36: 0x000000010003162c python`function_call + 364
    frame #37: 0x000000010000b853 python`PyObject_Call + 99
    frame #38: 0x0000000100018af7 python`instancemethod_call + 247
    frame #39: 0x000000010000b853 python`PyObject_Call + 99
    frame #40: 0x00000001000af05d python`PyEval_CallObjectWithKeywords + 93
    frame #41: 0x0000000100016aad python`PyInstance_New + 141
    frame #42: 0x000000010000b853 python`PyObject_Call + 99
    frame #43: 0x00000001000acbc7 python`PyEval_EvalFrameEx + 18567
    frame #44: 0x00000001000af946 python`fast_function + 342
    frame #45: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #46: 0x00000001000af946 python`fast_function + 342
    frame #47: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #48: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #49: 0x00000001000af866 python`fast_function + 118
    frame #50: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #51: 0x00000001000af946 python`fast_function + 342
    frame #52: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #53: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #54: 0x000000010003162c python`function_call + 364
    frame #55: 0x000000010000b853 python`PyObject_Call + 99
    frame #56: 0x0000000100018af7 python`instancemethod_call + 247
    frame #57: 0x000000010000b853 python`PyObject_Call + 99
    frame #58: 0x00000001000687d5 python`slot_tp_call + 133
    frame #59: 0x000000010000b853 python`PyObject_Call + 99
    frame #60: 0x00000001000ab65c python`PyEval_EvalFrameEx + 13084
    frame #61: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #62: 0x00000001000af866 python`fast_function + 118
    frame #63: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #64: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #65: 0x000000010003162c python`function_call + 364
    frame #66: 0x000000010000b853 python`PyObject_Call + 99
    frame #67: 0x0000000100018af7 python`instancemethod_call + 247
    frame #68: 0x000000010000b853 python`PyObject_Call + 99
    frame #69: 0x00000001000af05d python`PyEval_CallObjectWithKeywords + 93
    frame #70: 0x0000000100016aad python`PyInstance_New + 141
    frame #71: 0x000000010000b853 python`PyObject_Call + 99
    frame #72: 0x00000001000acbc7 python`PyEval_EvalFrameEx + 18567
    frame #73: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #74: 0x000000010003162c python`function_call + 364
    frame #75: 0x000000010000b853 python`PyObject_Call + 99
    frame #76: 0x00000001000ab65c python`PyEval_EvalFrameEx + 13084
    frame #77: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #78: 0x00000001000af866 python`fast_function + 118
    frame #79: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #80: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #81: 0x00000001000af866 python`fast_function + 118
    frame #82: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #83: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #84: 0x000000010003162c python`function_call + 364
    frame #85: 0x000000010000b853 python`PyObject_Call + 99
    frame #86: 0x00000001000ab65c python`PyEval_EvalFrameEx + 13084
    frame #87: 0x00000001000af946 python`fast_function + 342
    frame #88: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #89: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #90: 0x000000010003162c python`function_call + 364
    frame #91: 0x000000010000b853 python`PyObject_Call + 99
    frame #92: 0x0000000100018af7 python`instancemethod_call + 247
    frame #93: 0x000000010000b853 python`PyObject_Call + 99
    frame #94: 0x00000001000af05d python`PyEval_CallObjectWithKeywords + 93
    frame #95: 0x0000000100016aad python`PyInstance_New + 141
    frame #96: 0x000000010000b853 python`PyObject_Call + 99
    frame #97: 0x00000001000acbc7 python`PyEval_EvalFrameEx + 18567
    frame #98: 0x00000001000af946 python`fast_function + 342
    frame #99: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #100: 0x00000001000af946 python`fast_function + 342
    frame #101: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #102: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #103: 0x00000001000af866 python`fast_function + 118
    frame #104: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #105: 0x00000001000af946 python`fast_function + 342
    frame #106: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #107: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #108: 0x000000010003162c python`function_call + 364
    frame #109: 0x000000010000b853 python`PyObject_Call + 99
    frame #110: 0x0000000100018af7 python`instancemethod_call + 247
    frame #111: 0x000000010000b853 python`PyObject_Call + 99
    frame #112: 0x00000001000687d5 python`slot_tp_call + 133
    frame #113: 0x000000010000b853 python`PyObject_Call + 99
    frame #114: 0x00000001000acbc7 python`PyEval_EvalFrameEx + 18567
    frame #115: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #116: 0x000000010003162c python`function_call + 364
    frame #117: 0x000000010000b853 python`PyObject_Call + 99
    frame #118: 0x00000001000ab65c python`PyEval_EvalFrameEx + 13084
    frame #119: 0x00000001000af946 python`fast_function + 342
    frame #120: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #121: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #122: 0x00000001000af866 python`fast_function + 118
    frame #123: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #124: 0x00000001000af946 python`fast_function + 342
    frame #125: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #126: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #127: 0x000000010003162c python`function_call + 364
    frame #128: 0x000000010000b853 python`PyObject_Call + 99
    frame #129: 0x0000000100018af7 python`instancemethod_call + 247
    frame #130: 0x000000010000b853 python`PyObject_Call + 99
    frame #131: 0x00000001000687d5 python`slot_tp_call + 133
    frame #132: 0x000000010000b853 python`PyObject_Call + 99
    frame #133: 0x00000001000acbc7 python`PyEval_EvalFrameEx + 18567
    frame #134: 0x00000001000af946 python`fast_function + 342
    frame #135: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #136: 0x00000001000af946 python`fast_function + 342
    frame #137: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #138: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #139: 0x000000010003162c python`function_call + 364
    frame #140: 0x000000010000b853 python`PyObject_Call + 99
    frame #141: 0x00000001000ab65c python`PyEval_EvalFrameEx + 13084
    frame #142: 0x00000001000af946 python`fast_function + 342
    frame #143: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #144: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #145: 0x00000001000af866 python`fast_function + 118
    frame #146: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #147: 0x00000001000af946 python`fast_function + 342
    frame #148: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #149: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #150: 0x000000010003162c python`function_call + 364
    frame #151: 0x000000010000b853 python`PyObject_Call + 99
    frame #152: 0x0000000100018af7 python`instancemethod_call + 247
    frame #153: 0x000000010000b853 python`PyObject_Call + 99
    frame #154: 0x00000001000687d5 python`slot_tp_call + 133
    frame #155: 0x000000010000b853 python`PyObject_Call + 99
    frame #156: 0x00000001000acbc7 python`PyEval_EvalFrameEx + 18567
    frame #157: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #158: 0x00000001000af866 python`fast_function + 118
    frame #159: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #160: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #161: 0x00000001000ac746 python`PyEval_EvalFrameEx + 17414
    frame #162: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #163: 0x00000001000af866 python`fast_function + 118
    frame #164: 0x00000001000aba2f python`PyEval_EvalFrameEx + 14063
    frame #165: 0x00000001000a7ee2 python`PyEval_EvalCodeEx + 1666
    frame #166: 0x000000010003162c python`function_call + 364
    frame #167: 0x000000010000b853 python`PyObject_Call + 99
    frame #168: 0x00000001000e6fb1 python`RunModule + 113
    frame #169: 0x00000001000e6ac1 python`Py_Main + 2465
    frame #170: 0x00007fff960b05ad libdyld.dylib`start + 1

Help us @reaperhulk you’re our only hope.

[reaperhulk]

I can replicate this with the current cryptography master on my Mac. No idea why we're not seeing this on our CI though. This is the vector that causes the failure (and this script fails with a bus error on my Mac)

from cryptography.hazmat.backends.openssl.backend import backend
from cryptography.hazmat.primitives.asymmetric import ec


vector = {'y': 17502091976454499611505625879374774917623246385936044091564633236054L, 'x': 1075547452530705099852971289727135188477809665711500674593974877078L, 'curve': 'secp224r1', 'd': 24409883047820789832454338490434427874387169505509737608417835723568L}

numbers = ec.EllipticCurvePrivateNumbers(
    vector['d'],
    ec.EllipticCurvePublicNumbers(
        vector['x'],
        vector['y'],
        ec.SECP224R1()
    )
)
key = numbers.private_key(backend)

[hynek]

FWIW, it doesn’t crash on my local Ubuntu with OpenSSL 1.0.1f too. Do you think that’s a bug in 1.0.2 on OS X?

[reaperhulk]

It only happens with universal builds of OpenSSL, so it's definitely OS X only. I don't have a good explanation for why this occurs right now though.

EC_GFp_nistp224_method is an x86-64 specific constant time function, so it is suggestive that when linking against a universal library (which contains both i386 and x86-64 data) it crashes, while the x86-64 only library exhibits no such problem.

[reaperhulk]

Figured it out. Here we go...

Cryptography binds functions that may or may not be available. Unfortunately, cffi provides us no method by which we can say "include this symbol if present, otherwise leave it out". Instead we need to call cdef to declare our types and functions, but we don't actually know if those functions exist until we invoke the compiler/linker. To handle this we developed a pattern whereby we use the C preprocessor to introspect our OpenSSL (via the headers) and, if symbols are missing, we declare them as null function pointers (or other values if they're not functions) and set a flag. During import of the Python module we then iterate over the set of conditional symbols and remove all attributes that aren't present from the lib object.

In this particular case we declared a flag Cryptography_HAS_EC_NISTP_64_GCC_128 and then used it in this conditional:

#if defined(OPENSSL_NO_EC) || OPENSSL_VERSION_NUMBER < 0x1000100f || \
    defined(OPENSSL_NO_EC_NISTP_64_GCC_128)
static const long Cryptography_HAS_EC_NISTP_64_GCC_128 = 0;

const EC_METHOD *(*EC_GFp_nistp224_method)(void) = NULL;
const EC_METHOD *(*EC_GFp_nistp256_method)(void) = NULL;
const EC_METHOD *(*EC_GFp_nistp521_method)(void) = NULL;
#else
static const long Cryptography_HAS_EC_NISTP_64_GCC_128 = 1;
#endif

So far so good. The problem arises when building a universal library on OS X and then linking against the static version of it. To understand that we need a quick digression into what a universal library is on OS X...

Universal libraries are simply multiple CPU architectures crammed together in one file (typically using a tool called lipo). The linker simply looks at what arch it is linking for and uses those while ignoring the others. Universal libraries typically have i386 and x86_64 symbols in them these days, but just a few years ago you could find ppc, ppc64, i386, and x86_64 inside some libs.

When compiling cryptography (or any project) against a universal lib you specify the headers and then the library location (if they're not already in your include and lib search paths) and then the compiler does the rest. Unfortunately, universal libraries apparently assume that the headers will be the same between architectures. This is not true for any project that leaks build flags into the public headers. In OpenSSL's case this comes through with OPENSSL_NO_EC_NISTP_64_GCC_128. The above code declares EC_GFp_nistp224_method (and friends) as NULL function pointers if OPENSSL_NO_EC_NISTP_64_GCC_128 is set. Unfortunately, while this is set on the i386 build, it is not set on x86_64. Homebrew, when building a universal library, chooses to keep the headers from the i386 build. So, when cryptography compiles the C preprocessor dutifully says that OPENSSL_NO_EC_NISTP_64_GCC_128 is set and therefore EC_GFp_nistp224_method, etc are not present and should be defined as NULL function pointers.

Of course, this should still be caught right? After all, the ec.h header contains a different declaration for this function and we explicitly include the headers so that any mismatch will cause a compiler error! Let's look at the header file!

# ifndef OPENSSL_NO_EC_NISTP_64_GCC_128
/** Returns 64-bit optimized methods for nistp224
 *  \return  EC_METHOD object
 */
const EC_METHOD *EC_GFp_nistp224_method(void);

/** Returns 64-bit optimized methods for nistp256
 *  \return  EC_METHOD object
 */
const EC_METHOD *EC_GFp_nistp256_method(void);

/** Returns 64-bit optimized methods for nistp521
 *  \return  EC_METHOD object
 */
const EC_METHOD *EC_GFp_nistp521_method(void);
# endif

😭

But wait, even if this all gets compiled the symbol is still present in the libcrypto.a right? Why isn't that called? I'm still not clear on that one but for now it's ✨linker reasons✨. Similarly, this doesn't occur if you dynamically link rather than statically link: ✨linker reasons✨.

Finally, if we're removing the function conditionally why is it being called and failing? Well, internally EC_GROUP_new_by_curve_name (and a few other functions) call it. Since the x86_64 build was compiled with EC_NISTP_64_GCC_128 support they believe the function is available, but when they call it they get the null function pointer and it crashes. This is from the C layer, so the removal of the Python attribute is not relevant.

Summary and What We're Gonna Do™

So, when running in 64-bit mode linked against a universal OpenSSL built using homebrew the current cryptography release (and master) will SIGBUS due to a header mismatch during compilation causing a symbol to be improperly defined as a NULL function pointer. This affects many scenarios involving elliptic curve keys on OS X right now, although the failure mode (crash!) is at least mostly safe.

The functions in question have no need to be bound at all (they're really internal implementation details and are of no benefit to us), so dropping them from the bindings will resolve this issue. Going forward, I'm going to propose altering our NULL function pointer approach to instead be something more like

 const EC_METHOD *EC_GFp_nistp224_method(void) {
     printf("EC_GFp_nistp224_method was called. Aborting!\n");
     abort();
 }

rather than

const EC_METHOD *(*EC_GFp_nistp224_method)(void) = NULL;

[Lukasa]

👏 👏 👏

This bug is stupid and @reaperhulk is awesome for catching it. Well done!

[hynek]

This has been fixed by the amazing @reaperhulk.