OpenSSL 3.0.0: password callback called multiple times

[tiran]

BPO	43789
Nosy	@tiran, @miss-islington
PRs	bpo-43789: OpenSSL 3.0.0 Don't call passwd callback again in error case (GH-25303) #25303 [3.9] bpo-43789: OpenSSL 3.0.0 Don't call passwd callback again in error case (GH-25303) #25305 [3.8] bpo-43789: OpenSSL 3.0.0 Don't call passwd callback again in error case (GH-25303) (GH-25306) #25306

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tiran'
closed_at = None
created_at = <Date 2021-04-09.12:06:06.407>
labels = ['expert-SSL', 'type-bug', '3.8', '3.9', '3.10']
title = 'OpenSSL 3.0.0: password callback called multiple times'
updated_at = <Date 2021-04-09.14:56:32.329>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2021-04-09.14:56:32.329>
actor = 'christian.heimes'
assignee = 'christian.heimes'
closed = False
closed_date = None
closer = None
components = ['SSL']
creation = <Date 2021-04-09.12:06:06.407>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 43789
keywords = ['patch']
message_count = 5.0
messages = ['390608', '390619', '390622', '390633', '390634']
nosy_count = 2.0
nosy_names = ['christian.heimes', 'miss-islington']
pr_nums = ['25303', '25305', '25306']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43789'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[tiran]

OpenSSL 3.0.0 seems to invoke the password callback multiple times under some circumstances. This triggers a fatal error in Python when the first invocation sets an exception.

test_load_cert_chain (test.test_ssl.ContextTests) ... Fatal Python error: _PyEval_EvalFrameDefault: a function returned a result with an exception set
Python runtime state: initialized
ValueError: password cannot be longer than 1023 bytes

Current thread 0x00007fa88787f740 (most recent call first):
File "/home/heimes/dev/python/cpython/Lib/test/test_ssl.py", line 1395 in getpass_huge
File "/home/heimes/dev/python/cpython/Lib/test/test_ssl.py", line 1415 in test_load_cert_chain
File "/home/heimes/dev/python/cpython/Lib/unittest/case.py", line 549 in _callTestMethod
File "/home/heimes/dev/python/cpython/Lib/unittest/case.py", line 592 in run
File "/home/heimes/dev/python/cpython/Lib/unittest/case.py", line 652 in __call__
File "/home/heimes/dev/python/cpython/Lib/unittest/suite.py", line 122 in run
File "/home/heimes/dev/python/cpython/Lib/unittest/suite.py", line 84 in __call__
File "/home/heimes/dev/python/cpython/Lib/unittest/suite.py", line 122 in run
File "/home/heimes/dev/python/cpython/Lib/unittest/suite.py", line 84 in __call__
File "/home/heimes/dev/python/cpython/Lib/unittest/runner.py", line 176 in run
File "/home/heimes/dev/python/cpython/Lib/test/support/init.py", line 959 in _run_suite
File "/home/heimes/dev/python/cpython/Lib/test/support/init.py", line 1082 in run_unittest
File "/home/heimes/dev/python/cpython/Lib/test/test_ssl.py", line 4836 in test_main
File "/home/heimes/dev/python/cpython/Lib/test/libregrtest/runtest.py", line 246 in _runtest_inner2
File "/home/heimes/dev/python/cpython/Lib/test/libregrtest/runtest.py", line 282 in _runtest_inner
File "/home/heimes/dev/python/cpython/Lib/test/libregrtest/runtest.py", line 154 in _runtest
File "/home/heimes/dev/python/cpython/Lib/test/libregrtest/runtest.py", line 194 in runtest
File "/home/heimes/dev/python/cpython/Lib/test/libregrtest/main.py", line 321 in rerun_failed_tests
File "/home/heimes/dev/python/cpython/Lib/test/libregrtest/main.py", line 698 in _main
File "/home/heimes/dev/python/cpython/Lib/test/libregrtest/main.py", line 641 in main
File "/home/heimes/dev/python/cpython/Lib/test/libregrtest/main.py", line 719 in main
File "/home/heimes/dev/python/cpython/Lib/test/main.py", line 2 in <module>
File "/home/heimes/dev/python/cpython/Lib/runpy.py", line 86 in _run_code
File "/home/heimes/dev/python/cpython/Lib/runpy.py", line 196 in _run_module_as_main

Extension modules: _testcapi (total: 1)

(gdb) bt
#0 0x00007ffff7c5d9d5 in raise () from /lib64/libc.so.6
#1 0x00007ffff7c468a4 in abort () from /lib64/libc.so.6
#2 0x000000000051bb53 in fatal_error_exit (status=<optimized out>) at Python/pylifecycle.c:2522
#3 0x000000000051f97e in fatal_error (fd=2, header=header@entry=1, prefix=prefix@entry=0x6c2f60 <func.47> "_PyEval_EvalFrameDefault",
msg=msg@entry=0x670aa8 "a function returned a result with an exception set", status=status@entry=-1) at Python/pylifecycle.c:2703
#4 0x000000000051f9df in _Py_FatalErrorFunc (func=func@entry=0x6c2f60 <func.47> "_PyEval_EvalFrameDefault",
msg=msg@entry=0x670aa8 "a function returned a result with an exception set") at Python/pylifecycle.c:2719
#5 0x00000000004d930c in _PyEval_EvalFrameDefault (tstate=0x807060,
f=Frame 0x7fffe950e5b0, for file /home/heimes/dev/python/cpython/Lib/test/test_ssl.py, line 1395, in getpass_huge (), throwflag=0) at Python/ceval.c:1733
#6 0x00000000004e640f in _PyEval_EvalFrame (throwflag=0,
f=Frame 0x7fffe950e5b0, for file /home/heimes/dev/python/cpython/Lib/test/test_ssl.py, line 1395, in getpass_huge (), tstate=0x807060)
at ./Include/internal/pycore_ceval.h:46
#7 _PyEval_Vector (tstate=0x807060, con=0x7fffe9377c30, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>)
at Python/ceval.c:5109
#8 0x000000000042bf08 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:342
#9 0x00007fffe9cf502c in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=0, args=0x0, callable=<function at remote 0x7fffe9377c20>, tstate=0x807060)
at ./Include/cpython/abstract.h:114
#10 _PyObject_CallNoArg (func=<function at remote 0x7fffe9377c20>) at ./Include/cpython/abstract.h:168
#11 _password_callback (buf=0x7fffffff80c0 "p", size=1023, rwflag=<optimized out>, userdata=0x7fffffff9820) at /home/heimes/dev/python/cpython/Modules/_ssl.c:3935
#12 0x00007fffe9a3bcd2 in ui_read (ui=0xc22d70, uis=0xba6190) at crypto/ui/ui_util.c:111
#13 0x00007fffe9a3a4e0 in UI_process (ui=0xc22d70) at crypto/ui/ui_lib.c:516
#14 0x00007fffe99a3d49 in do_ui_passphrase (pass=0x7fffffff87b0 "", pass_size=1024, pass_len=0x7fffffff8bb8, prompt_info=0x0, verify=0, ui_method=0xc20050,
ui_data=0x7fffffff9820) at crypto/passphrase.c:173
#15 0x00007fffe99a4143 in ossl_pw_get_passphrase (pass=0x7fffffff87b0 "", pass_size=1024, pass_len=0x7fffffff8bb8, params=0x0, verify=0, data=0xc21cc8)
at crypto/passphrase.c:269
#16 0x00007fffe99a43ca in ossl_pw_passphrase_callback_dec (pass=0x7fffffff87b0 "", pass_size=1024, pass_len=0x7fffffff8bb8, params=0x0, arg=0xc21cc8)
at crypto/passphrase.c:324
#17 0x00007fffe99a3f8a in ossl_pw_get_passphrase (pass=0x7fffffff87b0 "", pass_size=1024, pass_len=0x7fffffff8bb8, params=0x0, verify=0, data=0xc0d0c8)
at crypto/passphrase.c:231
#18 0x00007fffe99a43ca in ossl_pw_passphrase_callback_dec (pass=0x7fffffff87b0 "", pass_size=1024, pass_len=0x7fffffff8bb8, params=0x0, arg=0xc0d0c8)
at crypto/passphrase.c:324
#19 0x00007fffe9a9d5ca in der2key_decode_p8 (input_der=0x7fffffff8d50, input_der_len=1905, ctx=0xbb2c40, pw_cb=0x7fffe99a4389 <ossl_pw_passphrase_callback_dec>,
pw_cbarg=0xc0d0c8, key_from_pkcs8=0x7fffe99e8653 <ossl_rsa_key_from_pkcs8>) at providers/implementations/encode_decode/decode_der2key.c:150
#20 0x00007fffe9a9e432 in rsa_d2i_PKCS8 (key=0x0, der=0x7fffffff8d50, der_len=1905, ctx=0xbb2c40, pw_cb=0x7fffe99a4389 <ossl_pw_passphrase_callback_dec>, pw_cbarg=0xc0d0c8)
at providers/implementations/encode_decode/decode_der2key.c:580
#21 0x00007fffe9a9da9d in der2key_decode (vctx=0xbb2c40, cin=0xc4d0d0, selection=135, data_cb=0x7fffe9931e91 <decoder_process>, data_cbarg=0x7fffffff8de0,
pw_cb=0x7fffe99a4389 <ossl_pw_passphrase_callback_dec>, pw_cbarg=0xc0d0c8) at providers/implementations/encode_decode/decode_der2key.c:295
#22 0x00007fffe993240b in decoder_process (params=0x0, arg=0x7fffffff8f00) at crypto/encode_decode/decoder_lib.c:750
#23 0x00007fffe9930f03 in OSSL_DECODER_from_bio (ctx=0xc0d090, in=0xc09700) at crypto/encode_decode/decoder_lib.c:58
#24 0x00007fffe99310b6 in OSSL_DECODER_from_data (ctx=0xc0d090, pdata=0x7fffffff8fc0, pdata_len=0x7fffffff8fb8) at crypto/encode_decode/decoder_lib.c:108
#25 0x00007fffe9a2fc77 in try_key_value (data=0x7fffffff9060, ctx=0xc21c80, cb=0x7fffe99a4389 <ossl_pw_passphrase_callback_dec>, cbarg=0xc21cc8,
libctx=0x7fffe9c1f880 <default_context_int>, propq=0x0) at crypto/store/store_result.c:288
#26 0x00007fffe9a301cc in try_key (data=0x7fffffff9060, v=0x7fffffff96c0, ctx=0xc21c80, provider=0x97f7d0, libctx=0x7fffe9c1f880 <default_context_int>, propq=0x0)
at crypto/store/store_result.c:407
#27 0x00007fffe9a2f6e6 in ossl_store_handle_load_result (params=0x7fffffff92f0, arg=0x7fffffff96c0) at crypto/store/store_result.c:152
#28 0x00007fffe9ac4638 in file_load_construct (decoder_inst=0x88b0a0, params=0x7fffffff92f0, construct_data=0x7fffffff9640)
at providers/implementations/storemgmt/file_store.c:402
#29 0x00007fffe9931fd6 in decoder_process (params=0x7fffffff92f0, arg=0x7fffffff94b0) at crypto/encode_decode/decoder_lib.c:566
#30 0x00007fffe9aa03a8 in pem2der_decode (vctx=0xbb6ef0, cin=0xc04d70, selection=0, data_cb=0x7fffe9931e91 <decoder_process>, data_cbarg=0x7fffffff94b0,
pw_cb=0x7fffe99a4389 <ossl_pw_passphrase_callback_dec>, pw_cbarg=0xbc14f8) at providers/implementations/encode_decode/decode_pem2der.c:243
#31 0x00007fffe993240b in decoder_process (params=0x0, arg=0x7fffffff95d0) at crypto/encode_decode/decoder_lib.c:750
#32 0x00007fffe9930f03 in OSSL_DECODER_from_bio (ctx=0xbc14c0, in=0xc28ab0) at crypto/encode_decode/decoder_lib.c:58
#33 0x00007fffe9ac49db in file_load_file (ctx=0xc282b0, object_cb=0x7fffe9a2f3c2 <ossl_store_handle_load_result>, object_cbarg=0x7fffffff96c0,
pw_cb=0x7fffe99a4389 <ossl_pw_passphrase_callback_dec>, pw_cbarg=0xc21cc8) at providers/implementations/storemgmt/file_store.c:522
--Type <RET> for more, q to quit, c to continue without paging--
#34 0x00007fffe9ac4f6f in file_load (loaderctx=0xc282b0, object_cb=0x7fffe9a2f3c2 <ossl_store_handle_load_result>, object_cbarg=0x7fffffff96c0,
pw_cb=0x7fffe99a4389 <ossl_pw_passphrase_callback_dec>, pw_cbarg=0xc21cc8) at providers/implementations/storemgmt/file_store.c:682
#35 0x00007fffe9a2c3ff in OSSL_STORE_load (ctx=0xc21c80) at crypto/store/store_lib.c:386
#36 0x00007fffe99c48d0 in pem_read_bio_key (bp=0xbb6050, x=0x0, cb=0x7fffe9cf4f99 <_password_callback>, u=0x7fffffff9820, libctx=0x0, propq=0x0,
expected_store_info_type=4, try_secure=1) at crypto/pem/pem_pkey.c:74
#37 0x00007fffe99c4b12 in PEM_read_bio_PrivateKey_ex (bp=0xbb6050, x=0x0, cb=0x7fffe9cf4f99 <_password_callback>, u=0x7fffffff9820, libctx=0x0, propq=0x0)
at crypto/pem/pem_pkey.c:144
#38 0x00007fffe9c653e7 in SSL_CTX_use_PrivateKey_file (ctx=0xc28350, file=0x7fffea13a060 "/home/heimes/dev/python/cpython/Lib/test/keycert.passwd.pem", type=1)
at ssl/ssl_rsa.c:372
#39 0x00007fffe9cf82ce in _ssl__SSLContext_load_cert_chain_impl (self=self@entry=0x7fffe94c9be0, certfile='/home/heimes/dev/python/cpython/Lib/test/keycert.passwd.pem',
keyfile=0x0, password=<optimized out>) at /home/heimes/dev/python/cpython/Modules/_ssl.c:4032

[tiran]

New changeset d3b73f3 by Christian Heimes in branch 'master':
bpo-43789: OpenSSL 3.0.0 Don't call passwd callback again in error case (GH-25303)
d3b73f3

[miss-islington]

New changeset a188bd4 by Miss Islington (bot) in branch '3.9':
bpo-43789: OpenSSL 3.0.0 Don't call passwd callback again in error case (GH-25303)
a188bd4

[tiran]

New changeset a28398e by Miss Islington (bot) in branch '3.8':
[3.8] bpo-43789: OpenSSL 3.0.0 Don't call passwd callback again in error case (GH-25303) (GH-25306)
a28398e

[tiran]

I'm keeping the bug open as a reminder to investigate the change of behavior more carefully.

[erlend-aasland]

I'm keeping the bug open as a reminder to investigate the change of behavior more carefully.

This change has been with us since 3.10. I think we can close this issue now.