[BUG]: Heap-use-after-free in test_functions.py::test29_traceback

[NAThompson]

Problem description

Build a python3.11 interpreter via:

 Python-3.11.4 $ ./configure --with-address-sanitizer --with-undefined-behavior-sanitizer --with-openssl=$(brew --prefix openssl) --prefix=/some/dir
 Python-3.11.4 $ make -j `nproc`
 Python-3.11.4 $ make install

Create a venv and make sure that cmake finds this interpreter, then run the nanobind unit tests. Then run:

n/tests ❯❯❯ python3 -m pytest test_functions.py::test29_traceback -s -v 
====================================================== test session starts =======================================================
platform darwin -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0 ---
cachedir: .pytest_cache
configfile: pyproject.toml
collected 1 item

test_functions.py::test29_traceback =================================================================
==2380==ERROR: AddressSanitizer: heap-use-after-free on address 0x00010c1f3e70 at pc 0x00010cfaf498 bp 0x00016f02e9e0 sp 0x00016f02e9d8
READ of size 8 at 0x00010c1f3e70 thread T0
    #0 0x10cfaf494 in Py_DECREF(_object*) object.h:537
    #1 0x10cfa8854 in Py_XDECREF(_object*) object.h:602
    #2 0x10cfa8600 in nanobind::python_error::~python_error() error.cpp:82
    #3 0x10cfa8944 in nanobind::python_error::~python_error() error.cpp:74
    #4 0x1813e2e44 in __cxa_decrement_exception_refcount+0x38 (libc++abi.dylib:arm64e+0x17e44) (BuildId: 52aa13e2567c36c294947b892fdbf24532000000200000000100000000040d00)
    #5 0x10cef9f10 in nanobind_init_test_functions_ext(nanobind::module_&)::$_53::operator()(nanobind::callable) const test_functions.cpp:189
    #6 0x10cef8b04 in _object* nanobind::detail::func_create<false, true, nanobind_init_test_functions_ext(nanobind::module_&)::$_53, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, nanobind::callable, 0ul, nanobind::scope, nanobind::name>(nanobind_init_test_functions_ext(nanobind::module_&)::$_53&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> (*)(nanobind::callable), std::__1::integer_sequence<unsigned long, 0ul>, nanobind::scope const&, nanobind::name const&)::'lambda'(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*)::__invoke(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*) nb_func.h:105
    #7 0x10cf29bf4 in nanobind::detail::nb_func_vectorcall_simple(_object*, _object* const*, unsigned long, _object*) nb_func.cpp:736
    #8 0x100e97b38 in PyObject_Vectorcall call.c:299

0x00010c1f3e70 is located 16 bytes inside of 56-byte region [0x00010c1f3e60,0x00010c1f3e98)
freed by thread T0 here:
    #0 0x102046fa4 in wrap_free+0x98 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x42fa4) (BuildId: f0a7ac5c49bc3abc851181b6f92b308a32000000200000000100000000000b00)
    #1 0x1011b4920 in tb_dealloc traceback.c:175
    #2 0x100ec2eb8 in BaseException_clear exceptions.c:88
    #3 0x100ec2a40 in BaseException_dealloc exceptions.c:102
    #4 0x10cfaf524 in Py_DECREF(_object*) object.h:538
    #5 0x10cfa8854 in Py_XDECREF(_object*) object.h:602
    #6 0x10cfa84b0 in nanobind::python_error::~python_error() error.cpp:80
    #7 0x10cfa8944 in nanobind::python_error::~python_error() error.cpp:74
    #8 0x1813e2e44 in __cxa_decrement_exception_refcount+0x38 (libc++abi.dylib:arm64e+0x17e44) (BuildId: 52aa13e2567c36c294947b892fdbf24532000000200000000100000000040d00)
    #9 0x10cef9f10 in nanobind_init_test_functions_ext(nanobind::module_&)::$_53::operator()(nanobind::callable) const test_functions.cpp:189
    #10 0x10cef8b04 in _object* nanobind::detail::func_create<false, true, nanobind_init_test_functions_ext(nanobind::module_&)::$_53, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, nanobind::callable, 0ul, nanobind::scope, nanobind::name>(nanobind_init_test_functions_ext(nanobind::module_&)::$_53&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> (*)(nanobind::callable), std::__1::integer_sequence<unsigned long, 0ul>, nanobind::scope const&, nanobind::name const&)::'lambda'(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*)::__invoke(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*) nb_func.h:105
    #11 0x10cf29bf4 in nanobind::detail::nb_func_vectorcall_simple(_object*, _object* const*, unsigned long, _object*) nb_func.cpp:736
    #12 0x100e97b38 in PyObject_Vectorcall call.c:299
previously allocated by thread T0 here:
    #0 0x102046e68 in wrap_malloc+0x94 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x42e68) (BuildId: f0a7ac5c49bc3abc851181b6f92b308a32000000200000000100000000000b00)
    #1 0x1011db468 in _PyObject_GC_New gcmodule.c:2298
    #2 0x1011b5304 in PyTraceBack_Here traceback.c:253
    #3 0x1010b0560 in _PyEval_EvalFrameDefault ceval.c:5757
    #4 0x1010afa8c in _PyEval_Vector ceval.c:6439
    #5 0x100e97b38 in PyObject_Vectorcall call.c:299
    #6 0x10cf94c78 in nanobind::detail::obj_vectorcall(_object*, _object* const*, unsigned long, _object*, bool) common.cpp:306
    #7 0x10cefbe10 in nanobind::object nanobind::detail::api<nanobind::handle>::operator()<(nanobind::rv_policy)1>() const nb_call.h:136
    #8 0x10cef9d18 in nanobind_init_test_functions_ext(nanobind::module_&)::$_53::operator()(nanobind::callable) const test_functions.cpp:186
    #9 0x10cef8b04 in _object* nanobind::detail::func_create<false, true, nanobind_init_test_functions_ext(nanobind::module_&)::$_53, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, nanobind::callable, 0ul, nanobind::scope, nanobind::name>(nanobind_init_test_functions_ext(nanobind::module_&)::$_53&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> (*)(nanobind::callable), std::__1::integer_sequence<unsigned long, 0ul>, nanobind::scope const&, nanobind::name const&)::'lambda'(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*)::__invoke(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*) nb_func.h:105
    #10 0x10cf29bf4 in nanobind::detail::nb_func_vectorcall_simple(_object*, _object* const*, unsigned long, _object*) nb_func.cpp:736
    #11 0x100e97b38 in PyObject_Vectorcall call.c:299
SUMMARY: AddressSanitizer: heap-use-after-free object.h:537 in Py_DECREF(_object*)
Shadow bytes around the buggy address:
  0x00702185e770: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x00702185e780: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x00702185e790: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x00702185e7a0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x00702185e7b0: fa fa fa fa 00 00 00 00 00 00 00 01 fa fa fa fa
=>0x00702185e7c0: fd fd fd fd fd fd fd fd fa fa fa fa fd fd[fd]fd
  0x00702185e7d0: fd fd fd fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x00702185e7e0: fa fa fa fa fd fd fd fd fd fd fd fa fa fa fa fa
  0x00702185e7f0: fd fd fd fd fd fd fd fa fa fa fa fa fd fd fd fd
  0x00702185e800: fd fd fd fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x00702185e810: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07

Reproducible example code

Code is in the repo-only build config is necessary to reproduce.

[wjakob]

Thank you @NAThompson, that's a serious issue you discovered.

Compiling Python with sanitizers wasn't enough to reproduce the error on my end— I had to add +set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fsanitize=undefined,address") to the nanobind CMakeLists.txt file as well.

Fixed in 30d30ca.
I also found another unrelated (minor, it seems?) issue and fixed it here: c48b180.

[wjakob]

For posterity, this issue was introduced in nanobind version 1.5.0 (this commit: 36751cb)

[NAThompson]

@wjakob : Yes-I forgot to add the instruction to also compile nanobind with sanitizers as well. Will try to be more careful next time . . .

Sincerest thanks for the quick fix.