Closed
Description
Describe the issue
Build x64 debug version from main with tests enabled and they fail due to segfaults. Tested with VS 17.14.4 and 17.14.3. This is a regression from up to 4 months ago.
Urgency
Blocking us from updating to the latest ORT
Target platform
Windows x64
Build script
onnxruntime/tools/ci_build/build.py --config Debug --cmake_path "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe" --ctest_path "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\ctest.exe" --cmake_generator "Visual Studio 17 2022" --build_shared_lib --parallel 32 --build_dir C:\work\onnxruntime\build_x64-cpu --cmake_extra_defines "onnxruntime_USE_NEURAL_SPEED=OFF" --enable_msvc_static_runtime --use_binskim_compliant_compile_flags
Error / output
The following tests FAILED:
1 - onnxruntime_test_all (SEGFAULT)
2 - onnx_test_pytorch_converted (SEGFAULT)
3 - onnx_test_pytorch_operator (SEGFAULT)
4 - onnxruntime_shared_lib_dlopen_test (SEGFAULT)
5 - onnxruntime_shared_lib_test (SEGFAULT)
6 - onnxruntime_global_thread_pools_test (SEGFAULT)
7 - onnxruntime_customopregistration_test (SEGFAULT)
8 - onnxruntime_autoep_test (SEGFAULT)
9 - onnxruntime_logging_apis_test (SEGFAULT)
This seems to be due to #24542
Call stack at point of read access violation:
> std::ctype<char>::widen(char _Byte) Line 2755 C++
std::basic_ios<char,std::char_traits<char>>::widen(char _Byte) Line 113 C++
std::basic_ios<char,std::char_traits<char>>::init(std::basic_streambuf<char,std::char_traits<char>> * _Strbuf, bool _Isstd) Line 149 C++
std::basic_ostream<char,std::char_traits<char>>::basic_ostream<char,std::char_traits<char>>(std::basic_streambuf<char,std::char_traits<char>> * _Strbuf, bool _Isstd) Line 31 C++
std::basic_ostringstream<char,std::char_traits<char>,std::allocator<char>>::basic_ostringstream<char,std::char_traits<char>,std::allocator<char>>() Line 742 C++
Memory_LeakCheck::~Memory_LeakCheck() Line 221 C++
`dynamic atexit destructor for 'g_memory_leak_check''() C++
_execute_onexit_table::__l2::<lambda>() Line 206 C++
__crt_seh_guarded_call<int>::operator()<void <lambda>(void),int <lambda>(void) &,void <lambda>(void)>(__acrt_lock_and_call::__l2::void <lambda>(void) && setup, _execute_onexit_table::__l2::int <lambda>(void) & action, __acrt_lock_and_call::__l2::void <lambda>(void) && cleanup) Line 204 C++
__acrt_lock_and_call<int <lambda>(void)>(const __acrt_lock_id lock_id, _execute_onexit_table::__l2::int <lambda>(void) && action) Line 983 C++
_execute_onexit_table(_onexit_table_t * table) Line 231 C++
common_exit::__l2::<lambda>() Line 225 C++
__crt_seh_guarded_call<void>::operator()<void <lambda>(void),void <lambda>(void) &,void <lambda>(void)>(__acrt_lock_and_call::__l2::void <lambda>(void) && setup, common_exit::__l2::void <lambda>(void) & action, __acrt_lock_and_call::__l2::void <lambda>(void) && cleanup) Line 224 C++
__acrt_lock_and_call<void <lambda>(void)>(const __acrt_lock_id lock_id, common_exit::__l2::void <lambda>(void) && action) Line 983 C++
common_exit(const int return_code, const _crt_exit_cleanup_mode cleanup_mode, const _crt_exit_return_mode return_mode) Line 259 C++
exit(int return_code) Line 302 C++
__scrt_common_main_seh() Line 295 C++
__scrt_common_main() Line 331 C++
mainCRTStartup(void * __formal) Line 17 C++
BaseThreadInitThunk(unsigned long RunProcessInit, long(*)(void *) StartAddress, void * Argument) Line 77 C
RtlUserThreadStart(long(*)(void *) StartAddress, void * Argument) Line 1184 C
It looks like std::ctype<char> is in an invalid state. Presumably because the process is exiting.
Visual Studio Version
No response
GCC / Compiler Version
No response