Skip to content
This repository has been archived by the owner on Jan 16, 2024. It is now read-only.

AVS-SDK 1.14.0 Process exists (or crashes) when Multiple Barge-ins are done and internet connectivity is Lost #1468

Closed
indrachatterjee86 opened this issue Aug 16, 2019 · 14 comments

Comments

@indrachatterjee86
Copy link

Briefly summarize your issue:

With AVS-SDK 1.14.0, it is observed that AVS SDK process exists or crashes in between user query session.

What is the expected behavior?

Alexa Process should be alive and should not be killed.

What behavior are you observing?

Process Crash or Exist.

Provide the steps to reproduce the issue, if applicable:

  1. Alexa is configured.
  2. User to Say Alexa, and alexa would go to listening mode , ask query, then while Speaking state is in progress , barge-in with wake word and ask ALEXA another query.
  3. Keep doing the above in loop, and the process would exit.
  4. Multiple wake word invocations one after the other.

Tell us about your environment:

AVS SDK Version 1.14.0 Cross Compiled to run on below Hardware.

  1. PulseAudio
  2. Bluetooth

Tell us what hardware you're using:

Other: Hardware: Samsung Artik 305 Secure Board
ARH: ARMv7

Tell us about your OS (Type & version):

Linux, Ubuntu 16.04
Kernel : Linux artik 4.4.71-UNRELEASED #1 SMP PREEMPT Wed Aug 14 19:29:31 IST 2019 armv7l armv7l armv7l GNU/Linux

Debug Logs (Attached)

alexa.log

@scotthea-amazon
Copy link
Contributor

Hello @indrachatterjee86,

Thank you for bringing this to our attention and providing the logs. We will try and reproduce this locally.

It would probably be very helpful if you would post a stack backtrace for this crash.

Thank you again,
-SWH

@indrachatterjee86
Copy link
Author

Hello @scotthea-amazon
I tried to generate a core dump file , using command "ulimit -c unlimited" , before starting the AVS-SDK Process, but on crash i am not able to see any core dump file.

Is there any other way that i can generate the stack trace on my machine.

NOTE: I do not have enough space on the device to install gdb and attach a debugger while process is running.

@kclchan
Copy link
Contributor

kclchan commented Aug 20, 2019

Hi @indrachatterjee86, just want to confirm if you executed "ulimit -c unlimited" in the same console that you run the SampleApp with?

@mohitbansal19
Copy link

Hi @kclchan, we are starting alexa from a start script and this script executed on board boot up. "ulimit -c unlimited" command is executed on ssh session of our board.

@indrachatterjee86
Copy link
Author

Hello @kclchan , We have a system which has a read only root file system. Hence, Core file might not have been generated. To reproduce this, we have added the command "ulimit -c unlimited" just above the alexa binary execution command.

I shall add the core stack trace if i am able to get the same.

@oscarbailey-xmos
Copy link

oscarbailey-xmos commented Sep 3, 2019

We've been seeing the same issue in our fork of 1.14 https://github.com/xmos/avs-device-sdk/
The backtrace is as follows:

(gdb) bt
#0  0x7691a9a0 in std::__shared_ptr<alexaClientSDK::capabilityAgents::audioPlayer::AudioPlayer::PlayDirectiveInfo, (__gnu_cxx::_Lock_policy)1>::get (this=0x8) at /usr/include/c++/6/bits/shared_ptr_base.h:1063
#1  0x768ac5b4 in alexaClientSDK::capabilityAgents::audioPlayer::AudioPlayer::<lambda()>::operator()(void) const (__closure=0x72081ecc)
    at /home/pi/sdk-folder/avs-device-sdk/CapabilityAgents/AudioPlayer/src/AudioPlayer.cpp:200
#2  0x7690e368 in std::_Bind<alexaClientSDK::capabilityAgents::audioPlayer::AudioPlayer::cancelDirective(std::shared_ptr<alexaClientSDK::avsCommon::avs::CapabilityAgent::DirectiveInfo>)::<lambda()>()>::__call<void>(<unknown type in /home/pi/sdk-folder/sdk-build/CapabilityAgents/AudioPlayer/src/libAudioPlayer.so, CU 0x0, DIE 0x8f18f>, std::_Index_tuple<>) (this=0x72081ecc, 
    __args=<unknown type in /home/pi/sdk-folder/sdk-build/CapabilityAgents/AudioPlayer/src/libAudioPlayer.so, CU 0x0, DIE 0x8f18f>) at /usr/include/c++/6/functional:934
#3  0x7690d76c in std::_Bind<alexaClientSDK::capabilityAgents::audioPlayer::AudioPlayer::cancelDirective(std::shared_ptr<alexaClientSDK::avsCommon::avs::CapabilityAgent::DirectiveInfo>)::<lambda()>()>::operator()<>(void) (
    this=0x72081ecc) at /usr/include/c++/6/functional:993
#4  0x7690c7ec in std::__invoke_impl<void, std::_Bind<alexaClientSDK::capabilityAgents::audioPlayer::AudioPlayer::cancelDirective(std::shared_ptr<alexaClientSDK::avsCommon::avs::CapabilityAgent::DirectiveInfo>)::<lambda()>()>&>(std::__invoke_other, std::_Bind<alexaClientSDK::capabilityAgents::audioPlayer::AudioPlayer::cancelDirective(std::shared_ptr<alexaClientSDK::avsCommon::avs::CapabilityAgent::DirectiveInfo>)::<lambda()>()> &) (__f=...)
    at /usr/include/c++/6/functional:210
#5  0x7690b794 in std::__invoke<std::_Bind<alexaClientSDK::capabilityAgents::audioPlayer::AudioPlayer::cancelDirective(std::shared_ptr<alexaClientSDK::avsCommon::avs::CapabilityAgent::DirectiveInfo>)::<lambda()>()>&>(std::_Bind<alexaClientSDK::capabilityAgents::audioPlayer::AudioPlayer::cancelDirective(std::shared_ptr<alexaClientSDK::avsCommon::avs::CapabilityAgent::DirectiveInfo>)::<lambda()>()> &) (__fn=...) at /usr/include/c++/6/functional:251
#6  0x7690a6dc in std::reference_wrapper<std::_Bind<alexaClientSDK::capabilityAgents::audioPlayer::AudioPlayer::cancelDirective(std::shared_ptr<alexaClientSDK::avsCommon::avs::CapabilityAgent::DirectiveInfo>)::<lambda()>()> >::operator()<>(void) const (this=0x58eadbb0) at /usr/include/c++/6/functional:465
#7  0x76908cb4 in std::_Bind_simple<std::reference_wrapper<std::_Bind<alexaClientSDK::capabilityAgents::audioPlayer::AudioPlayer::cancelDirective(std::shared_ptr<alexaClientSDK::avsCommon::avs::CapabilityAgent::DirectiveInfo>)::<lambda()>()> >()>::_M_invoke<>(std::_Index_tuple<>) (this=0x58eadbb0) at /usr/include/c++/6/functional:1391
#8  0x76905fc0 in std::_Bind_simple<std::reference_wrapper<std::_Bind<alexaClientSDK::capabilityAgents::audioPlayer::AudioPlayer::cancelDirective(std::shared_ptr<alexaClientSDK::avsCommon::avs::CapabilityAgent::DirectiveInfo>)::<lambda()>()> >()>::operator()(void) (this=0x58eadbb0) at /usr/include/c++/6/functional:1380
#9  0x76903bf4 in std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::_Bind_simple<std::reference_wrapper<std::_Bind<alexaClientSDK::capabilityAgents::audioPlayer::AudioPlayer::cancelDirective(std::shared_ptr<alexaClientSDK::avsCommon::avs::CapabilityAgent::DirectiveInfo>)::<lambda()>()> >()>, void>::operator()(void) const (this=0x58eadbb8)
    at /usr/include/c++/6/future:1348
#10 0x768ffc00 in std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>(), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::_Bind_simple<std::reference_wrapper<std::_Bind<alexaClientSDK::capabilityAgents::audioPlayer::AudioPlayer::cancelDirective(std::shared_ptr<alexaClientSDK::avsCommon::avs::CapabilityAgent::DirectiveInfo>)::<lambda()>()> >()>, void> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/6/functional:1717
#11 0x001e3d74 in std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const (this=0x58eadbb8) at /usr/include/c++/6/functional:2127
#12 0x001e21d8 in std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) (this=0x72081eb4, __f=0x58eadbb8, 
    __did_set=0x58eadb67) at /usr/include/c++/6/future:533
#13 0x001e5e80 in std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (__f=<unknown type in /home/pi/sdk-folder/sdk-build/SampleApp/src/SampleApp, CU 0x36c2b, DIE 0xac9d9>, 
    __t=<unknown type in /home/pi/sdk-folder/sdk-build/SampleApp/src/SampleApp, CU 0x36c2b, DIE 0xab3db>, __args#0=<unknown type in /home/pi/sdk-folder/sdk-build/SampleApp/src/SampleApp, CU 0x36c2b, DIE 0xac9e5>, 
    __args#1=<unknown type in /home/pi/sdk-folder/sdk-build/SampleApp/src/SampleApp, CU 0x36c2b, DIE 0xac9f1>) at /usr/include/c++/6/functional:227
#14 0x001e505c in std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (__fn=<unknown type in /home/pi/sdk-folder/sdk-build/SampleApp/src/SampleApp, CU 0x36c2b, DIE 0xac9d9>, __args#0=<unknown type in /home/pi/sdk-folder/sdk-build/SampleApp/src/SampleApp, CU 0x36c2b, DIE 0xab3db>, 
    __args#1=<unknown type in /home/pi/sdk-folder/sdk-build/SampleApp/src/SampleApp, CU 0x36c2b, DIE 0xac9e5>, __args#2=<unknown type in /home/pi/sdk-folder/sdk-build/SampleApp/src/SampleApp, CU 0x36c2b, DIE 0xac9f1>)
    at /usr/include/c++/6/functional:251
#15 0x001e35f4 in void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}::operator()() const (__closure=0x58eadb28) at /usr/include/c++/6/mutex:602
#16 0x001e36bc in void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::operator()() const (__closure=0x0) at /usr/include/c++/6/mutex:607
#17 0x001e36dc in void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::_FUN() () at /usr/include/c++/6/mutex:607
#18 0x76c63128 in __pthread_once_slow (once_control=0x72081ec4, init_routine=0x153c7c <__once_proxy@plt>) at pthread_once.c:116
#19 0x00166880 in __gthread_once (__once=0x72081ec4, __func=0x153c7c <__once_proxy@plt>) at /usr/include/arm-linux-gnueabihf/c++/6/bits/gthr-default.h:699
#20 0x001e379c in std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (__once=..., __f=<unknown type in /home/pi/sdk-folder/sdk-build/SampleApp/src/SampleApp, CU 0x36c2b, DIE 0xac9d9>, 
---Type <return> to continue, or q <return> to quit---
    __args#0=<unknown type in /home/pi/sdk-folder/sdk-build/SampleApp/src/SampleApp, CU 0x36c2b, DIE 0xab3db>, __args#1=<unknown type in /home/pi/sdk-folder/sdk-build/SampleApp/src/SampleApp, CU 0x36c2b, DIE 0xac9e5>, 
    __args#2=<unknown type in /home/pi/sdk-folder/sdk-build/SampleApp/src/SampleApp, CU 0x36c2b, DIE 0xac9f1>) at /usr/include/c++/6/mutex:614
#21 0x001e1bf8 in std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) (this=0x72081eb4, __res=..., 
    __ignore_failure=false) at /usr/include/c++/6/future:393
#22 0x768f6484 in std::__future_base::_Task_state<std::_Bind<alexaClientSDK::capabilityAgents::audioPlayer::AudioPlayer::cancelDirective(std::shared_ptr<alexaClientSDK::avsCommon::avs::CapabilityAgent::DirectiveInfo>)::<lambda()>()>, std::allocator<int>, void()>::_M_run(void) (this=0x72081eb4) at /usr/include/c++/6/future:1409
#23 0x001e6490 in std::packaged_task<void ()>::operator()() (this=0x72081e84) at /usr/include/c++/6/future:1553
#24 0x768bac98 in alexaClientSDK::avsCommon::utils::threading::Executor::<lambda()>::operator()(void) (__closure=0x72003fe8)
    at /home/pi/sdk-folder/avs-device-sdk/AVSCommon/Utils/include/AVSCommon/Utils/Threading/Executor.h:213
#25 0x768d1d94 in std::_Function_handler<void(), alexaClientSDK::avsCommon::utils::threading::Executor::pushTo(bool, Task, Args&& ...) [with Task = alexaClientSDK::capabilityAgents::audioPlayer::AudioPlayer::cancelDirective(std::shared_ptr<alexaClientSDK::avsCommon::avs::CapabilityAgent::DirectiveInfo>)::<lambda()>; Args = {}; decltype (task(alexaClientSDK::avsCommon::utils::threading::Executor::pushTo::args ...)) = void]::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/6/functional:1731
#26 0x76abf4c8 in std::function<void ()>::operator()() const (this=0x58eadc38) at /usr/include/c++/6/functional:2127
#27 0x74d7679c in alexaClientSDK::avsCommon::utils::threading::Executor::runNext (this=0x511de8) at /home/pi/sdk-folder/avs-device-sdk/AVSCommon/Utils/src/Executor.cpp:68
#28 0x001ea758 in std::__invoke_impl<bool, bool (alexaClientSDK::avsCommon::utils::threading::Executor::* const&)(), alexaClientSDK::avsCommon::utils::threading::Executor*&> (
    __f=@0x72005fe8: (bool (alexaClientSDK::avsCommon::utils::threading::Executor::*)(alexaClientSDK::avsCommon::utils::threading::Executor * const)) 0x154d5c <alexaClientSDK::avsCommon::utils::threading::Executor::runNext()@plt>, __t=@0x72005ff0: 0x511de8) at /usr/include/c++/6/functional:227
#29 0x001e9fe0 in std::__invoke<bool (alexaClientSDK::avsCommon::utils::threading::Executor::* const&)(), alexaClientSDK::avsCommon::utils::threading::Executor*&> (
    __fn=@0x72005fe8: (bool (alexaClientSDK::avsCommon::utils::threading::Executor::*)(alexaClientSDK::avsCommon::utils::threading::Executor * const)) 0x154d5c <alexaClientSDK::avsCommon::utils::threading::Executor::runNext()@plt>, __args#0=@0x72005ff0: 0x511de8) at /usr/include/c++/6/functional:251
#30 0x001e963c in std::_Mem_fn_base<bool (alexaClientSDK::avsCommon::utils::threading::Executor::*)(), true>::operator()<alexaClientSDK::avsCommon::utils::threading::Executor*&> (this=0x72005fe8, 
    __args#0=@0x72005ff0: 0x511de8) at /usr/include/c++/6/functional:604
#31 0x001e8a30 in std::_Bind<std::_Mem_fn<bool (alexaClientSDK::avsCommon::utils::threading::Executor::*)()> (alexaClientSDK::avsCommon::utils::threading::Executor*)>::__call<bool, , 0u>(std::tuple<>&&, std::_Index_tuple<0u>)
    (this=0x72005fe8, __args=<unknown type in /home/pi/sdk-folder/sdk-build/SampleApp/src/SampleApp, CU 0x36c2b, DIE 0xb2aa8>) at /usr/include/c++/6/functional:934
#32 0x001e7db8 in std::_Bind<std::_Mem_fn<bool (alexaClientSDK::avsCommon::utils::threading::Executor::*)()> (alexaClientSDK::avsCommon::utils::threading::Executor*)>::operator()<, bool>() (this=0x72005fe8)
    at /usr/include/c++/6/functional:993

#33 0x001e72ec in std::_Function_handler<bool (), std::_Bind<std::_Mem_fn<bool (alexaClientSDK::avsCommon::utils::threading::Executor::*)()> (alexaClientSDK::avsCommon::utils::threading::Executor*)> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/include/c++/6/functional:1717
#34 0x74dccae8 in std::function<bool ()>::operator()() const (this=0x58eadd30) at /usr/include/c++/6/functional:2127
#35 0x74dcc278 in alexaClientSDK::avsCommon::utils::threading::TaskThread::run(std::function<bool ()>) (this=0x511e70, jobRunner=...) at /home/pi/sdk-folder/avs-device-sdk/AVSCommon/Utils/src/TaskThread.cpp:75
#36 0x74dcd948 in std::__invoke_impl<void, void (alexaClientSDK::avsCommon::utils::threading::TaskThread::* const&)(std::function<bool ()>), alexaClientSDK::avsCommon::utils::threading::TaskThread*&, std::function<bool ()>&>(std::__invoke_memfun_deref, void (alexaClientSDK::avsCommon::utils::threading::TaskThread::* const&)(std::function<bool ()>), alexaClientSDK::avsCommon::utils::threading::TaskThread*&, std::function<bool ()>&) (__f=
    @0x72005f5c: (void (alexaClientSDK::avsCommon::utils::threading::TaskThread::*)(alexaClientSDK::avsCommon::utils::threading::TaskThread * const, std::function<bool()>)) 0x74dcc1c0 <alexaClientSDK::avsCommon::utils::threading::TaskThread::run(std::function<bool ()>)>, __t=@0x72005f74: 0x511e70, __args#0=...) at /usr/include/c++/6/functional:227
#37 0x74dcd894 in std::__invoke<void (alexaClientSDK::avsCommon::utils::threading::TaskThread::* const&)(std::function<bool ()>), alexaClientSDK::avsCommon::utils::threading::TaskThread*&, std::function<bool ()>&>(void (alexaClientSDK::avsCommon::utils::threading::TaskThread::* const&)(std::function<bool ()>), alexaClientSDK::avsCommon::utils::threading::TaskThread*&, std::function<bool ()>&) (__fn=
    @0x72005f5c: (void (alexaClientSDK::avsCommon::utils::threading::TaskThread::*)(alexaClientSDK::avsCommon::utils::threading::TaskThread * const, std::function<bool()>)) 0x74dcc1c0 <alexaClientSDK::avsCommon::utils::threading::TaskThread::run(std::function<bool ()>)>, __args#0=@0x72005f74: 0x511e70, __args#1=...) at /usr/include/c++/6/functional:251
#38 0x74dcd81c in std::_Mem_fn_base<void (alexaClientSDK::avsCommon::utils::threading::TaskThread::*)(std::function<bool ()>), true>::operator()<alexaClientSDK::avsCommon::utils::threading::TaskThread*&, std::function<bool ()>&>(alexaClientSDK::avsCommon::utils::threading::TaskThread*&, std::function<bool ()>&) const (this=0x72005f5c, __args#0=@0x72005f74: 0x511e70, __args#1=...) at /usr/include/c++/6/functional:604
#39 0x74dcd770 in std::_Bind<std::_Mem_fn<void (alexaClientSDK::avsCommon::utils::threading::TaskThread::*)(std::function<bool ()>)> (alexaClientSDK::avsCommon::utils::threading::TaskThread*, std::function<bool ()>)>::__call<void, , 0u, 1u>(std::tuple<>&&, std::_Index_tuple<0u, 1u>) (this=0x72005f5c, __args=<unknown type in /home/pi/sdk-folder/sdk-build/AVSCommon/libAVSCommon.so, CU 0x493300, DIE 0x4a1398>) at /usr/include/c++/6/functional:934
#40 0x74dcd644 in std::_Bind<std::_Mem_fn<void (alexaClientSDK::avsCommon::utils::threading::TaskThread::*)(std::function<bool ()>)> (alexaClientSDK::avsCommon::utils::threading::TaskThread*, std::function<bool ()>)>::operator()<, void>() (this=0x72005f5c) at /usr/include/c++/6/functional:993
#41 0x74dcd614 in std::_Bind_simple<std::_Bind<std::_Mem_fn<void (alexaClientSDK::avsCommon::utils::threading::TaskThread::*)(std::function<bool ()>)> (alexaClientSDK::avsCommon::utils::threading::TaskThread*, std::function<bool ()>)> ()>::_M_invoke<>(std::_Index_tuple<>) (this=0x72005f5c) at /usr/include/c++/6/functional:1391
#42 0x74dcd580 in std::_Bind_simple<std::_Bind<std::_Mem_fn<void (alexaClientSDK::avsCommon::utils::threading::TaskThread::*)(std::function<bool ()>)> (alexaClientSDK::avsCommon::utils::threading::TaskThread*, std::function<bool ()>)> ()>::operator()() (this=0x72005f5c) at /usr/include/c++/6/functional:1380
#43 0x74dcd558 in std::thread::_State_impl<std::_Bind_simple<std::_Bind<std::_Mem_fn<void (alexaClientSDK::avsCommon::utils::threading::TaskThread::*)(std::function<bool ()>)> (alexaClientSDK::avsCommon::utils::threading::TaskThread*, std::function<bool ()>)> ()> >::_M_run() (this=0x72005f58) at /usr/include/c++/6/thread:197
#44 0x74a269dc in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#45 0x76c5afc4 in start_thread (arg=0x58eae420) at pthread_create.c:458
#46 0x7486e038 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:76 from /lib/arm-linux-gnueabihf/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) 

And running info threads gives:

(gdb) info threads
  Id   Target Id         Frame 
  1    Thread 0x73df9000 (LWP 1851) "SampleApp" 0x76c61d3c in __pthread_cond_timedwait (cond=0x350c40, mutex=0x350c24, abstime=0x7effea68) at pthread_cond_timedwait.c:200
  2    Thread 0x73df6420 (LWP 1855) "SampleApp" 0x7485e560 in read () at ../sysdeps/unix/syscall-template.S:84
  3    Thread 0x733aa420 (LWP 1856) "SampleApp" 0x74865120 in poll () at ../sysdeps/unix/syscall-template.S:84
  4    Thread 0x729ff420 (LWP 1857) "SampleApp" 0x74865120 in poll () at ../sysdeps/unix/syscall-template.S:84
  5    Thread 0x71fff420 (LWP 1858) "SampleApp" 0x74865120 in poll () at ../sysdeps/unix/syscall-template.S:84
  6    Thread 0x715ff420 (LWP 1859) "SampleApp" 0x74865120 in poll () at ../sysdeps/unix/syscall-template.S:84
  7    Thread 0x70dfe420 (LWP 1860) "SampleApp" 0x74865120 in poll () at ../sysdeps/unix/syscall-template.S:84
  8    Thread 0x701ff420 (LWP 1861) "SampleApp" 0x74865120 in poll () at ../sysdeps/unix/syscall-template.S:84
  9    Thread 0x6f9fe420 (LWP 1862) "SampleApp" 0x76c61d3c in __pthread_cond_timedwait (cond=0x500a48, mutex=0x500998, abstime=0x6f9fda10) at pthread_cond_timedwait.c:200
  12   Thread 0x6dffd420 (LWP 1865) "SampleApp" 0x76c61d3c in __pthread_cond_timedwait (cond=0x504618, mutex=0x504648, abstime=0x6dffcb88) at pthread_cond_timedwait.c:200
  13   Thread 0x6d7fc420 (LWP 1866) "SampleApp" 0x76c619a4 in __pthread_cond_wait (cond=0x505278, mutex=0x50522c) at pthread_cond_wait.c:188
  16   Thread 0x6b6f7420 (LWP 1869) "SampleApp" 0x76c619a4 in __pthread_cond_wait (cond=0x5098b8, mutex=0x5098a0) at pthread_cond_wait.c:188
  18   Thread 0x6aef6420 (LWP 1871) "SampleApp" 0x76c619a4 in __pthread_cond_wait (cond=0x504460, mutex=0x5043cc) at pthread_cond_wait.c:188
  19   Thread 0x6a6f5420 (LWP 1872) "SampleApp" 0x76c619a4 in __pthread_cond_wait (cond=0x50a970, mutex=0x50a890) at pthread_cond_wait.c:188
  29   Thread 0x666ed420 (LWP 1883) "SampleApp" 0x76c61d3c in __pthread_cond_timedwait (cond=0x72169018, mutex=0x72169048, abstime=0x666ec1b0) at pthread_cond_timedwait.c:200
  30   Thread 0x65eec420 (LWP 1884) "SampleApp" 0x74865120 in poll () at ../sysdeps/unix/syscall-template.S:84
  44   Thread 0x636e7420 (LWP 1898) "SampleApp" 0x74865120 in poll () at ../sysdeps/unix/syscall-template.S:84
  45   Thread 0x63ee8420 (LWP 1899) "SampleApp" 0x76c61d3c in __pthread_cond_timedwait (cond=0x72002a20, mutex=0x72002a08, abstime=0x63ee70a0) at pthread_cond_timedwait.c:200
  115  Thread 0x6bef8420 (LWP 1979) "decodedQueue:sr" syscall () at ../sysdeps/unix/sysv/linux/arm/syscall.S:37
  210  Thread 0x6efff420 (LWP 2082) "SampleApp" clone () at ../sysdeps/unix/sysv/linux/arm/clone.S:61
  257  Thread 0x69ef4420 (LWP 2131) "SampleApp" 0x76c61d3c in __pthread_cond_timedwait (cond=0x50fcf0, mutex=0x50fd20, abstime=0x69ef3b88) at pthread_cond_timedwait.c:200
  258  Thread 0x586ad420 (LWP 2132) "SampleApp" clone () at ../sysdeps/unix/sysv/linux/arm/clone.S:61
  266  Thread 0x566a9420 (LWP 2141) "SampleApp" clone () at ../sysdeps/unix/sysv/linux/arm/clone.S:61
* 267  Thread 0x58eae420 (LWP 2143) "SampleApp" 0x7691a9a0 in std::__shared_ptr<alexaClientSDK::capabilityAgents::audioPlayer::AudioPlayer::PlayDirectiveInfo, (__gnu_cxx::_Lock_policy)1>::get (this=0x8)
    at /usr/include/c++/6/bits/shared_ptr_base.h:1063
  268  Thread 0x6c7fa420 (LWP 2144) "SampleApp" clone () at ../sysdeps/unix/sysv/linux/arm/clone.S:61
  269  Thread 0x61ee4420 (LWP 2145) "SampleApp" clone () at ../sysdeps/unix/sysv/linux/arm/clone.S:61
  270  Thread 0x57eac420 (LWP 2146) "SampleApp" clone () at ../sysdeps/unix/sysv/linux/arm/clone.S:61
(gdb)

We couldn't replicate the issue in 1.13 so we're rolling back to that version until this issue is fixed. It looks like the bug is related to the "pre-buffering" changes to AudioPlayer in 1.14 https://github.com/alexa/avs-device-sdk/releases/tag/v1.14

EDIT: Core dump: avssegfault_core.zip (~400MB uncompressed)

@az2022
Copy link

az2022 commented Sep 12, 2019

I just want to add to this thread as I am seeing similar issue of Segfault. I am compiling and running on a Pi3 and crashes occur around trigger detection as originally stated above. Have seen similar symptom in v1.13 SDK as well. After upgrading to v1.14 I still see crashes around trigger detection. I have added my bt if it helps:
(gdb) bt #0 0x76c5f418 in std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::_Bind_simple<std::reference_wrapper<std::_Bind<alexaClientSDK::capabilityAgents::audioPlayer::AudioPlayer::cancelDirective(std::shared_ptr<alexaClientSDK::avsCommon::avs::CapabilityAgent::DirectiveInfo>)::{lambda()#1} ()> > ()>, void> >::_M_invoke(std::_Any_data const&) () from /lib/avslibs/libAudioPlayer.so #1 0x00073288 in std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) () #2 0x76e3e0d4 in __pthread_once_slow (once_control=0x71f05874, init_routine=0x453f4 <__once_proxy@plt>) at pthread_once.c:116 #3 0x00073bec in void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) () #4 0x00073c68 in std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) () #5 0x76c60434 in std::__future_base::_Task_state<std::_Bind<alexaClientSDK::capabilityAgents::audioPlayer::AudioPlayer::cancelDirective(std::shared_ptr<alexaClientSDK::avsCommon::avs::CapabilityAgent::DirectiveInfo>)::{lambda()#1} ()>, std::allocator<int>, void ()>::_M_run() () from /lib/avslibs/libAudioPlayer.so #6 0x76c657cc in std::_Function_handler<void (), std::future<decltype ({parm#2}({parm#3}...))> alexaClientSDK::avsCommon::utils::threading::Executor::pushTo<alexaClientSDK::capabilityAgents::audioPlayer::AudioPlayer::cancelDirective(std::shared_ptr<alexaClientSDK::avsCommon::avs::CapabilityAgent::DirectiveInfo>)::{lambda()#1}>(bool, alexaClientSDK::capabilityAgents::audioPlayer::AudioPlayer::cancelDirective(std::shared_ptr<alexaClientSDK::avsCommon::avs::CapabilityAgent::DirectiveInfo>)::{lambda()#1})::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /lib/avslibs/libAudioPlayer.so #7 0x75d69970 in alexaClientSDK::avsCommon::utils::threading::Executor::runNext() () from /lib/avslibs/libAVSCommon.so #8 0x75da0780 in alexaClientSDK::avsCommon::utils::threading::TaskThread::run(std::function<bool ()>) () from /lib/avslibs/libAVSCommon.so #9 0x75da0e94 in std::thread::_State_impl<std::_Bind_simple<std::_Bind<std::_Mem_fn<void (alexaClientSDK::avsCommon::utils::threading::TaskThread::*)(std::function<bool ()>)> (alexaClientSDK::avsCommon::utils::threading::TaskThread*, std::function<bool ()>)> ()> >::_M_run() () from /lib/avslibs/libAVSCommon.so #10 0x75bbe9dc in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6 #11 0x76e35fc4 in start_thread (arg=0x67a5e420) at pthread_create.c:335 #12 0x75a07bc8 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:76 from /lib/arm-linux-gnueabihf/libc.so.6 Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb)

@az2022
Copy link

az2022 commented Sep 12, 2019

Any quick out-of-release updates that we can try is appreciated as the crashes have become a show-stopper for us. Thanks.

@kclchan
Copy link
Contributor

kclchan commented Sep 19, 2019

Hi @az2022 and @oscarbailey-xmos, thank you so much for the backtrace, it's extremely useful. Looking at the backtrace, the crash happened in AudioPlayer::cancelDirective where it uses the executor is trying to cleanup the m_preHandlePlayInfoList that matches the messageId. There is a bug in the for loop though as the iterator is invalidated after erase, and it++ changes the it to point to an invalid memory location, hence causing the crash.

Please follow the example code in https://en.cppreference.com/w/cpp/container/list/erase to change the logic in the for loop a bit to fix this issue. Thanks!

@az2022
Copy link

az2022 commented Sep 19, 2019

@kclchan - Thanks for getting back. Will give it a try. BTW, can you share when's the next target release of AVS-SDK please?

@az2022
Copy link

az2022 commented Sep 23, 2019

After applying the patch I am still seeing a Segfault but my backtrace doesn't show much this time:
(gdb) bt #0 0x76ec1404 in memcmp () from /usr/lib/arm-linux-gnueabihf/libarmmem.so #1 0x00000000 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb)

It appeared to die in Listening state:

###########################

Thinking...

###########################

##############################################################################

RenderTemplateCard

#-----------------------------------------------------------------------------

Focus State : FOREGROUND

Template Type : BodyTemplate2

Main Title : Who wrote walden?

##############################################################################

###########################

Speaking...

###########################

########################################

Alexa is currently idle!

########################################

##############################################################################

RenderTemplateCard - Cleared

##############################################################################

############################

Listening...

############################

###########################

Thinking...

###########################

##############################################################################

RenderTemplateCard

#-----------------------------------------------------------------------------

Focus State : FOREGROUND

Template Type : BodyTemplate2

Main Title : What is the capital of australia?

##############################################################################

###########################

Speaking...

###########################

############################

Listening...

############################

./alexa: line 12: 1116 Segmentation fault (core dumped) LD_LIBRARY_PATH=/lib/avslibs /lib/avslibs/SampleApp -C /home/pi/sdk-folder/AlexaClientSDKConfig.json -C /home/pi/sdk-folder/vteConfigSnsrAP.json

kclchan added a commit that referenced this issue Sep 25, 2019
Changes in this update:

**Enhancements**

* Added `SystemSoundPlayer` to [ApplicationUtilities](https://alexa.github.io/avs-device-sdk/namespacealexa_client_s_d_k_1_1application_utilities.html). `SystemSoundPlayer` is a new class that plays pre-defined sounds. Sounds currently supported include the wake word notification and the end of speech tone. This change is internal and you don't need to update your code.
* Removed [Echo Spatial Perception (ESP)](https://developer.amazon.com/blogs/alexa/post/042be85c-5a62-4c55-a18d-d7a82cf394df/esp-moves-to-the-cloud-for-alexa-enabled-devices) functionality from the Alexa Voice Service (AVS) device SDK. Make sure you download and test your devices using the new AVS SDK sample app. If you're using an older version of the sample app, manually remove any references to ESP or errors occur during compile.
* Added `onNotificationReceived` to `NotificationsObserverInterface`. `onNotificationReceived` broadcasts when `NotificationsObserverInterface` receives a new notification, instead of only sending the indicator state. This is important if you support a feature that requires a distinct signal for each notification received. See [NotificationsObserverInterface](https://alexa.github.io/avs-device-sdk/classalexa_client_s_d_k_1_1avs_common_1_1sdk_interfaces_1_1_notifications_observer_interface.html) for more details.
* Added support for [Multilingual Mode](https://developer.amazon.com/docs/alexa-voice-service/system.html#localecombinations). With this enabled, Alexa automatically detects what language a user speaks by analyzing the spoken wake word and proceeding utterances. Once Alexa identifies the language, all corresponding responses are in the same language. The current supported language pairs are:
 - `[ "en-US", "es-US" ]`
 - `[ "es-US", "en-US" ]`
 - `[ "en-IN", "hi-IN" ]`
 - `[ "hi-IN", "en-IN" ]`
 - `[ "en-CA", "fr-CA" ]`
 - `[ "fr-CA", "en-CA" ]`
<br/> **IMPORTANT**: Specify the locales your device supports in the [localeCombinations](https://developer.amazon.com/docs/alexa-voice-service/system.html#localecombinations) field in AlexaClientSDKConfig.json. This field can't be empty. If you don't set these values, the sample app fails to run.
* Added two new system settings, [Timezone](https://developer.amazon.com/docs/alexa-voice-service/system.html#settimezone) and [Locale](https://developer.amazon.com/docs/alexa-voice-service/system.html#locales).
    - Timezone: For example, you can set the `defaultTimezone` to `America/Vancouver`. If you don't set a value, `GMT` is set as the default value. If you set a new timezone, make sure that your AVS system settings and default timezone stay in sync. To handle this, use the new class `SystemTimeZoneInterface`. See [System Interface > SetTimeZone](https://developer.amazon.com/docs/alexa-voice-service/system.html#settimezone) for more information.
    - Locale: For example, you can set `defaultLocale` to `en-GB`, instead of the default `en-US`.
* The [SpeechRecognizer](https://developer.amazon.com/docs/alexa-voice-service/speechrecognizer.html) interface now supports the following functionalities.
  - Change wake word (`Alexa` supported for now).
  - Toggle start of request tone on/off.
  - Toggle End of request tone on/off.
* Deprecated the [CapabilityAgents](https://alexa.github.io/avs-device-sdk/classalexa_client_s_d_k_1_1avs_common_1_1avs_1_1_capability_agent.html) `Settings{…}` library. `Settings {…}` now maps to an interface that's no longer supported. You might need to update your code to handle these changes. Read [Settings Interface](https://developer.amazon.com/docs/alexa-voice-service/per-interface-settings.html) for more details.
* Added support for three new locals: Spanish - United States (ES_US), Hindi - India (HI_IN), and Brazilian - Portuguese (PT_BR).
* Linked the atomic library to the sample app to prevent build errors on Raspberry Pi.

**Bug Fixes**

* Fixed resource leaking in [EqualizerCapabilityAgent](https://alexa.github.io/avs-device-sdk/classalexa_client_s_d_k_1_1capability_agents_1_1equalizer_1_1_equalizer_capability_agent.html) after engine shutdown.
* [Issue 1391:](#1391) Fixed an issue where [SQLiteDeviceSettingsStorage::open](https://alexa.github.io/avs-device-sdk/classalexa_client_s_d_k_1_1settings_1_1storage_1_1_s_q_lite_device_setting_storage.html#a7733e56145916f7ff265c5c950add492) tries to acquire a mutex twice, resulting in deadlock.
* [Issue 1468:](#1468) Fixed a bug in [AudioPlayer::cancelDirective](https://alexa.github.io/avs-device-sdk/classalexa_client_s_d_k_1_1capability_agents_1_1audio_player_1_1_audio_player.html#a2c710c16f3627790fcc3238d34da9361) that causes a crash.
* Fixed Windows install script that caused the sample app build to fail - removed pip, flask, requests, and commentjson dependencies from the mingw.sh helper script.
* Fixed issue: notifications failed to sync upon device initialization. For example, let's say you had two devices - one turned on and the other turned off. After clearing the notification on the first device, it still showed up on the second device after turning it on.
* Fixed issue: barging in on a reminder caused it to stick in an inconsistent state, blocking subsequent reminders. For example, if a reminder was going off and you interrupted it, the reminder would get persist indefinitely. You could schedule future reminders, but they wouldn't play. Saying “Alexa stop” or rebooting the device fixed the “stuck” reminder.

**Known Issues**

* Music playback history isn't displayed in the Alexa app for certain account and device types.
* When using Gnu Compiler Collection 8+ (GCC 8+), `-Wclass-memaccess` triggers warnings. You can ignore these, they don't cause the build to fail.
* Android error `libDefaultClient.so not found` might occur. Resolve this by upgrading to ADB version 1.0.40.
* If a device loses a network connection, the lost connection status isn't returned via local TTS.
* ACL encounters issues if it receives audio attachments but doesn't consume them.
* `SpeechSynthesizerState` uses `GAINING_FOCUS` and `LOSING_FOCUS` as a  workaround for handling intermediate states.
* Media steamed through Bluetooth might abruptly stop. To restart playback, resume the media in the source application or toggle next/previous.
* If a connected Bluetooth device is inactive, the Alexa app might indicates that audio is playing.
* The Bluetooth agent assumes that the Bluetooth adapter is always connected to a power source. Disconnecting from a power source during operation isn't yet supported.
* When using some products, interrupted Bluetooth playback might not resume if other content is locally streamed.
* `make integration` isn't available for Android. To run Android integration tests, manually upload the test binary and input file and run ADB.
* Alexa might truncate the beginning of speech when responding to text-to-speech (TTS) user events. This only impacts Raspberry Pi devices running Android Things with HDMI output audio.
* A reminder TTS message doesn't play if the sample app restarts and loses a network connection. Instead, the default alarm tone plays twice.
* `ServerDisconnectIntegratonTest` tests are disabled until they are updated to reflect new service behavior.
* Bluetooth initialization must complete before connecting devices, otherwise devices are ignored.
* The `DirectiveSequencerTest.test_handleBlockingThenImmediatelyThenNonBockingOnSameDialogId` test fails intermittently.
@kclchan
Copy link
Contributor

kclchan commented Oct 18, 2019

Hi @az2022, without a backtrace, it is not possible to root cause it. The original issue is fixed in v1.15 of the SDK. Could you see if you can still reproduce the issue with this new version of the SDK?

@az2022
Copy link

az2022 commented Oct 24, 2019

Will try v1.15 SDK sometime next week. The trouble is difficulty reproducing this issue and when I see it the backtrace is wiped out! Thanks.

@caleighatamazon
Copy link
Contributor

I'm going to close this issue for now, since we believe it was fixed in a previous release. Please do re-open this issue if you are able to reproduce it using SDK 1.15 or 1.16. Thanks!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants