Is kaldi threadsafe when CUDA is enabled?

[chandlerz]

As the title, is kaldi threadsafe when CUDA is enabled?
I want to use kaldi for speech inference with multi CPU threads, but it crashes with following error and backtrace.

It runs normal with a single CPU thread or multi CPU threads without CUDA. Any suggestion to fix the issue?
Thanks!

---

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff94ff9700 (LWP 26231)]
0x0000000000689458 in std::_Hashtable<void*, std::pair<void* const, kaldi::CuMemoryAllocator::UsedMemoryElement>, std::allocator<std::pair<void* const, kaldi::CuMemoryAllocator::UsedMemoryElement> >, std::__detail::_Select1st, std::equal_to<void*>, kaldi::CuMemoryAllocator::PointerHasher, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_find_before_node(unsigned long, void* const&, unsigned long) const [clone .isra.163] ()
(gdb) backtrace
#0 0x0000000000689458 in std::_Hashtable<void*, std::pair<void* const, kaldi::CuMemoryAllocator::UsedMemoryElement>, std::allocator<std::pair<void* const, kaldi::CuMemoryAllocator::UsedMemoryElement> >, std::__detail::_Select1st, std::equal_to<void*>, kaldi::CuMemoryAllocator::PointerHasher, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_find_before_node(unsigned long, void* const&, unsigned long) const [clone .isra.163] ()
#1 0x000000000068adad in kaldi::CuMemoryAllocator::Free(void*) ()
#2 0x000000000069d3fd in kaldi::CuMatrix::Destroy() ()
#3 0x00000000006a438d in kaldi::CuMatrix::Resize(int, int, kaldi::MatrixResizeType, kaldi::MatrixStrideType) ()
#4 0x00000000006cd8b8 in kaldi::nnet3::NnetComputer::ExecuteCommand() ()
#5 0x00000000006cece2 in kaldi::nnet3::NnetComputer::Run() ()
#6 0x00000000006c3f0d in kaldi::nnet3::DecodableNnetLoopedOnlineBase::AdvanceChunk() ()
#7 0x00000000006c42a0 in kaldi::nnet3::DecodableAmNnetLoopedOnline::LogLikelihood(int, int) ()
#8 0x00000000004640fd in kaldi::LatticeFasterOnlineDecoder::ProcessEmitting(kaldi::DecodableInterface*) ()
#9 0x00000000004644db in kaldi::LatticeFasterOnlineDecoder::AdvanceDecoding(kaldi::DecodableInterface*, int) ()

[danpovey]

The memory allocation code for CUDA is not thread safe in the master branch. In the rnnlm branch that issue has been fixed, but you have to announce that you plan to use multiple threads by calling AllowMultithreading() on the CuDevice object. Dan

…

On Tue, Sep 5, 2017 at 7:29 PM, chandler zhou ***@***.***> wrote: As the title, is kaldi threadsafe when CUDA is enabled? I want to use kaldi for speech inference with multi CPU threads, but it crashes with following error and backtrace. It runs normal with a single CPU thread or multi CPU threads without CUDA. Any suggestion to fix the issue? Thanks! ------------------------------ Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fff94ff9700 (LWP 26231)] 0x0000000000689458 in std::_Hashtable<void*, std::pair<void* const, kaldi::CuMemoryAllocator::UsedMemoryElement>, std::allocator<std::pair<void* const, kaldi::CuMemoryAllocator::UsedMemoryElement> >, std::__detail::_Select1st, std::equal_to<void*>, kaldi::CuMemoryAllocator::PointerHasher, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_find_before_node(unsigned long, void* const&, unsigned long) const [clone .isra.163] () (gdb) backtrace #0 0x0000000000689458 in std::_Hashtable<void*, std::pair<void* const, kaldi::CuMemoryAllocator::UsedMemoryElement>, std::allocator<std::pair<void* const, kaldi::CuMemoryAllocator::UsedMemoryElement> >, std::__detail::_Select1st, std::equal_to<void*>, kaldi::CuMemoryAllocator::PointerHasher, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_find_before_node(unsigned long, void* const&, unsigned long) const [clone .isra.163] () #1 <#1> 0x000000000068adad in kaldi::CuMemoryAllocator::Free(void*) () #2 <#2> 0x000000000069d3fd in kaldi::CuMatrix::Destroy() () #3 <#3> 0x00000000006a438d in kaldi::CuMatrix::Resize(int, int, kaldi::MatrixResizeType, kaldi::MatrixStrideType) () #4 <#4> 0x00000000006cd8b8 in kaldi::nnet3::NnetComputer::ExecuteCommand() () #5 <#5> 0x00000000006cece2 in kaldi::nnet3::NnetComputer::Run() () #6 <#6> 0x00000000006c3f0d in kaldi::nnet3::DecodableNnetLoopedOnlineBase::AdvanceChunk() () #7 <#7> 0x00000000006c42a0 in kaldi::nnet3::DecodableAmNnetLoopedOnline::LogLikelihood(int, int) () #8 <#8> 0x00000000004640fd in kaldi::LatticeFasterOnlineDecoder::ProcessEmitting(kaldi::DecodableInterface*) () #9 <#9> 0x00000000004644db in kaldi::LatticeFasterOnlineDecoder::AdvanceDecoding(kaldi::DecodableInterface*, int) () #10 <#10> 0x00000000004166b7 in PaddyAsr::PaddyEngineImpl::ProcessTask(void*, char*, unsigned long) () #11 <#11> 0x00000000004151cc in PaddyAsr::PaddyEngine::ProcessTask(void*, char*, unsigned long) () #12 <#12> 0x0000000000414d57 in recog_one_task (task=0xd8d4a68, ***@***.***=1) at thread-decoder-noaff.cc:136 #13 <#13> 0x000000000041510a in work (args=0x7fffffffe360) at thread-decoder-noaff.cc:103 #14 <#14> 0x00007ffff79c4dc5 in start_thread () from /lib64/libpthread.so.0 #15 <#15> 0x00007fffef86873d in clone () from /lib64/libc.so.6 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1872>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADJVu2-m2Um5NyNrpD6DqW5ajeHfe5D9ks5sfgN0gaJpZM4PNwfX> .