Why am I getting a decryption error during encryption?

[jaraco]

In jaraco/jaraco.crypto#5, I may have stumbled onto a different manifestation of the issue reported in #21439.

In jaraco/jaraco.crypto#5, I've got one environment, my ARM macOS workstation running homebrew and OpenSSL 3.1.2 and 1.1.1v fails to encrypt using the exact same routine that passes on Windows, Linux, and macOS in Github actions and Windows ARM locally.

I've put together this standalone repro:

import ctypes
from ctypes import c_void_p, c_int, c_char, c_ulong

lib_path = '/opt/homebrew/opt/openssl@3/lib/libcrypto.dylib'
lib = ctypes.cdll.LoadLibrary(lib_path)

MAX_BLOCK_LENGTH = 32
MAX_IV_LENGTH = 16


class CipherType(ctypes.Structure):
    _fields_ = [
        ('nid', c_int),
        ('block_size', c_int),
        ('key_len', c_int),
        ('iv_len', c_int),
        ('flags', c_ulong),
        ('init', c_void_p),
        ('do_cipher', c_void_p),
        ('cleanup', c_void_p),
        ('ctx_size', c_int),
        ('set_asn1_parameters', c_void_p),
        ('get_asn1_parameters', c_void_p),
        ('ctrl', c_void_p),
        ('app_data', c_void_p),
    ]


class Cipher(ctypes.Structure):
    _fields_ = [
        ('cipher', c_void_p),  # POINTER(CipherType)
        ('engine', c_void_p),  # POINTER(ENGINE)
        ('encrypt', c_int),
        ('buf_len', c_int),
        ('oiv', c_char * MAX_IV_LENGTH),
        ('iv', c_char * MAX_IV_LENGTH),
        ('buf', c_char * MAX_BLOCK_LENGTH),
        ('num', c_int),
        ('app_data', c_void_p),
        ('key_len', c_int),
        ('flags', c_ulong),
        ('cipher_data', c_void_p),
        ('final_used', c_int),
        ('block_mask', c_int),
        ('final', c_char * MAX_BLOCK_LENGTH),
    ]


def get_error():
    err = lib.ERR_get_error()
    msg = ctypes.create_string_buffer(1024)
    lib.ERR_error_string(err, msg)
    return msg.value.decode()


lib.EVP_CIPHER_CTX_new.restype = ctypes.POINTER(Cipher)

ctx = lib.EVP_CIPHER_CTX_new().contents
assert ctx
key = '11111111111111111111111111111111'.encode()
iv = '1111111111111111'.encode()

engine = None
lib.EVP_get_cipherbyname.argtypes = (ctypes.c_char_p,)
lib.EVP_get_cipherbyname.restype = ctypes.POINTER(CipherType)
cipher = lib.EVP_get_cipherbyname('AES-256-CBC'.encode('ascii'))
assert cipher
encrypt = True
lib.EVP_CipherInit_ex.argtypes = (
    ctypes.POINTER(Cipher),
    ctypes.POINTER(CipherType),
    ctypes.c_void_p,
    ctypes.c_char_p,
    ctypes.c_char_p,
)

res = lib.EVP_CipherInit_ex(ctx, cipher.contents, engine, key, iv, encrypt)
if res == 0:
    print(f"Error initializing cipher: {get_error()}")
    raise SystemExit(1)

data = b''
out = ctypes.create_string_buffer(len(data) + MAX_BLOCK_LENGTH - 1)
out_len = ctypes.c_int()

out_data = []

lib.EVP_CipherUpdate.argtypes = (
    ctypes.POINTER(Cipher),
    ctypes.c_char_p,
    ctypes.POINTER(ctypes.c_int),
    ctypes.c_char_p,
    ctypes.c_int,
)

res = lib.EVP_CipherUpdate(ctx, out, out_len, data, len(data))
if res != 1:
    print(f"Error updating cipher: {get_error()}")
    raise SystemExit(1)

out_data.append(out.raw[: out_len.value])

out = ctypes.create_string_buffer(MAX_BLOCK_LENGTH)
out_len = ctypes.c_int()

lib.EVP_CipherFinal_ex.argtypes = (
    ctypes.POINTER(Cipher),
    ctypes.c_char_p,
    ctypes.POINTER(ctypes.c_int),
)
res = lib.EVP_CipherFinal_ex(ctx, out, out_len)
if res != 1:
    print(f"Error finalizing cipher: {get_error()}")

When run, it emits:

Error finalizing cipher: error:1C80006B:Provider routines::wrong final block length

When run on OpenSSL 1.1, it emits:

Error finalizing cipher: error:0606506D:digital envelope routines:EVP_DecryptFinal_ex:wrong final block length

Illustrating the "decrypt error during encrypt" behavior.

I tried turning on padding, but that didn't have any effect on the outcome.

It's currently hard-coded for openssl@3 on my mac, but you can change the library to a .so or .dll on Linux or Windows and it should work.

It's entirely possible I'm holding it wrong. Can someone help locate the fault? How is it that EVP_DecryptFinal_ex shows up when encrypt = True?

[jaraco]

I figured it out. I added some troubleshooting to check the value of ctx.encrypt and it was 0, even immediately after the initialization.

The issue is in the FFI spec for EVP_CipherInit_ex:

lib.EVP_CipherInit_ex.argtypes = (
    ctypes.POINTER(Cipher),
    ctypes.POINTER(CipherType),
    ctypes.c_void_p,
    ctypes.c_char_p,
    ctypes.c_char_p,
)

The spec is missing the last parameter for enc.

And for some reason that I don't yet understand, on Python for ARM for macOS only, ctypes either ignores the encrypt parameter or casts True to 0.

Adding ctypes.c_int to the signature resolves the issue.

[jaraco]

Root cause is with ctypes.