Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The 1.10 Android mystery crash thread! #13057

Closed
hrydgard opened this issue Jun 27, 2020 · 19 comments
Closed

The 1.10 Android mystery crash thread! #13057

hrydgard opened this issue Jun 27, 2020 · 19 comments
Milestone

Comments

@hrydgard
Copy link
Owner

Here we go again, analyzing new-looking crashes from Google Play to see if it's worth doing a 1.10.1 release.

First up is this ~Download crash:

42 minutes ago on app version 110000000
Huawei HUAWEI Y5 lite (HWDRA-MG), 1024MB RAM, Android 8.1
Report 1 of 3

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> org.ppsspp.ppsspp <<<

backtrace:
  #00  pc 000000000001a3ce  /system/lib/libc.so (abort+63)
  #01  pc 000000000001a5f1  /system/lib/libc.so (__assert2+20)
  #02  pc 0000000000a5ba67  /data/app/org.ppsspp.ppsspp-AwnAK9xgTrCKkKZOjrXMPA==/lib/arm/libppsspp_jni.so
  #03  pc 0000000000a5baef  /data/app/org.ppsspp.ppsspp-AwnAK9xgTrCKkKZOjrXMPA==/lib/arm/libppsspp_jni.so
  #04  pc 0000000000a5a129  /data/app/org.ppsspp.ppsspp-AwnAK9xgTrCKkKZOjrXMPA==/lib/arm/libppsspp_jni.so
  #05  pc 0000000000a5a0c1  /data/app/org.ppsspp.ppsspp-AwnAK9xgTrCKkKZOjrXMPA==/lib/arm/libppsspp_jni.so (std::terminate()+28)
  #06  pc 0000000000a58865  /data/app/org.ppsspp.ppsspp-AwnAK9xgTrCKkKZOjrXMPA==/lib/arm/libppsspp_jni.so (std::__ndk1::thread::~thread()+12)
  #07  pc 000000000053335f  /data/app/org.ppsspp.ppsspp-AwnAK9xgTrCKkKZOjrXMPA==/lib/arm/libppsspp_jni.so (http::Download::~Download()+152)
  #08  pc 000000000053489d  /data/app/org.ppsspp.ppsspp-AwnAK9xgTrCKkKZOjrXMPA==/lib/arm/libppsspp_jni.so (std::__ndk1::__shared_ptr_pointer<http::Download*, std::__ndk1::default_delete<http::Download>, std::__ndk1::allocator<http::Download>>::__on_zero_shared()+12)
  #09  pc 00000000005347bf  /data/app/org.ppsspp.ppsspp-AwnAK9xgTrCKkKZOjrXMPA==/lib/arm/libppsspp_jni.so (void* std::__ndk1::__thread_proxy<std::__ndk1::tuple<std::__ndk1::unique_ptr<std::__ndk1::__thread_struct, std::__ndk1::default_delete<std::__ndk1::__thread_struct> >, std::__ndk1::__bind<void (http::Download::*)(std::__ndk1::shared_ptr<http::Download>), http::Download*, std::__ndk1::shared_ptr<http::Download>&> > >(void*)+166)
  #10  pc 000000000004752f  /system/lib/libc.so (__pthread_start(void*)+22)
  #11  pc 000000000001af9d  /system/lib/libc.so (__start_thread+32)

This one feels like we still have some Vulkan mistake on shutdown, so not so critical (or it's just a driver bug, god knows there are enough of those):

27 minutes ago on app version 110000000
Huawei Y9 Prime 2019 (HWSTK-HF), 3840MB RAM, Android 10
Report 1 of 1
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> org.ppsspp.ppsspp <<<

backtrace:
  #00  pc 000000000006f06c  /apex/com.android.runtime/lib64/bionic/libc.so (abort+160)
  #01  pc 00000000000cf73c  /apex/com.android.runtime/lib64/bionic/libc.so (__fortify_fatal(char const*, ...)+116)
  #02  pc 00000000000cee70  /apex/com.android.runtime/lib64/bionic/libc.so (HandleUsingDestroyedMutex(pthread_mutex_t*, char const*)+52)
  #03  pc 00000000000cecd4  /apex/com.android.runtime/lib64/bionic/libc.so (pthread_mutex_lock+148)
  #04  pc 000000000245145c  /vendor/lib64/egl/libGLES_mali.so
  #05  pc 00000000024513c0  /vendor/lib64/egl/libGLES_mali.so
  #06  pc 000000000228a36c  /vendor/lib64/egl/libGLES_mali.so
  #07  pc 000000000228a124  /vendor/lib64/egl/libGLES_mali.so (vkQueueSubmit+80)
  #08  pc 000000000070e5e8  /data/app/org.ppsspp.ppsspp-4uJCUxd3fSg8xSVKpyIaoQ==/split_config.arm64_v8a.apk!libppsspp_jni.so (offset 0x1000) (VulkanRenderManager::Submit(int, bool)+816)
  #09  pc 000000000070e758  /data/app/org.ppsspp.ppsspp-4uJCUxd3fSg8xSVKpyIaoQ==/split_config.arm64_v8a.apk!libppsspp_jni.so (offset 0x1000) (VulkanRenderManager::EndSubmitFrame(int)+64)
  #10  pc 000000000070b2b4  /data/app/org.ppsspp.ppsspp-4uJCUxd3fSg8xSVKpyIaoQ==/split_config.arm64_v8a.apk!libppsspp_jni.so (offset 0x1000) (VulkanRenderManager::ThreadFunc()+368)
  #11  pc 000000000070ec68  /data/app/org.ppsspp.ppsspp-4uJCUxd3fSg8xSVKpyIaoQ==/split_config.arm64_v8a.apk!libppsspp_jni.so (offset 0x1000) (void* std::__ndk1::__thread_proxy<std::__ndk1::tuple<std::__ndk1::unique_ptr<std::__ndk1::__thread_struct, std::__ndk1::default_delete<std::__ndk1::__thread_struct> >, void (VulkanRenderManager::*)(), VulkanRenderManager*> >(void*)+64)
  #12  pc 00000000000ce1b0  /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+36)
  #13  pc 0000000000070ba8  /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64)

To be continued!

@hrydgard hrydgard added this to the v1.10.1 milestone Jun 27, 2020
@unknownbrackets
Copy link
Collaborator

For Download, we just do this:

	if (thread_.joinable())
		thread_.join();

I figured this was equivalent to converting it to a pointer and doing a nullcheck, but maybe there's some difference? As long as STL is behaving as advertised, this makes me worry about a double free or memory corruption?

-[Unknown]

@hrydgard
Copy link
Owner Author

I don't know if we maybe have to do if (thread && thread.joinable()) maybe? but ideally we shouldn't be in a position where thread can be "null" though anyway....

@unknownbrackets
Copy link
Collaborator

Yeah, we start both downloads immediately, so there should always be a thread. And nothing else joins it.

I think we still have exceptions off - I wonder what happens if the thread fails to start... but I mean, according to the trace thread was destructing after Download destructed, and STL should:

If *this has an associated thread (joinable() == true), std::terminate() is called.

And a postcondition of join(), which we call, is that joinable() becomes false.

-[Unknown]

@hrydgard
Copy link
Owner Author

Indeed, weird. Don't really have a lot of data yet (rollout at 1.5%), got 4 reports so far of it, 4 that got coalesced and 2 separate ones (only those two got auto-marked with NEW).

I'll take a look at the code myself tomorrow but I don't expect to figure out much more than you did...

Not seeing anything else dramatic so far. I'm gonna ramp up the rollout to 5%.

@hrydgard
Copy link
Owner Author

The Download thing continues to be the most common new crash.

This one in PPGe is the runner-up so far:

Acer Liquid S1 (a10), 1024MB RAM, Android 10
Report 1

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> org.ppsspp.ppsspp <<<

backtrace:
  #00  pc 00000000003cb582  /data/app/org.ppsspp.ppsspp-V6uDCI3e1gytN9BnXhsn2A==/split_config.armeabi_v7a.apk!libppsspp_jni.so (offset 0x1000)
  #01  pc 00000000003cb209  /data/app/org.ppsspp.ppsspp-V6uDCI3e1gytN9BnXhsn2A==/split_config.armeabi_v7a.apk!libppsspp_jni.so (offset 0x1000) (PPGeDrawText(char const*, float, float, PPGeAlign, float, unsigned int)+156)
  #02  pc 00000000002d2667  /data/app/org.ppsspp.ppsspp-V6uDCI3e1gytN9BnXhsn2A==/split_config.armeabi_v7a.apk!libppsspp_jni.so (offset 0x1000) (PSPSaveDialog::DisplaySaveDataInfo1()+1290)
  #03  pc 00000000002d3609  /data/app/org.ppsspp.ppsspp-V6uDCI3e1gytN9BnXhsn2A==/split_config.armeabi_v7a.apk!libppsspp_jni.so (offset 0x1000) (PSPSaveDialog::Update(int)+1956)
  #04  pc 0000000000387efb  /data/app/org.ppsspp.ppsspp-V6uDCI3e1gytN9BnXhsn2A==/split_config.armeabi_v7a.apk!libppsspp_jni.so (offset 0x1000)
  #05  pc 00000000002f7fef  /data/app/org.ppsspp.ppsspp-V6uDCI3e1gytN9BnXhsn2A==/split_config.armeabi_v7a.apk!libppsspp_jni.so (offset 0x1000) (CallSyscallWithoutFlags(HLEFunction const*)+14)
  #06  pc 0000000000484402  <anonymous>

@hrydgard
Copy link
Owner Author

I have a theory about the Download thing. Apparently we send in a shared_ptr to the thread, to keep "self" alive. But if that is the only owner of self.. the destruction seems to contain contradictions. It seems in that case the destructor would run on the thread itself when the thread function exits, and a thread joining itself doesn't sound healthy...

@unknownbrackets
Copy link
Collaborator

Oh, that's a good theory. Actually sounds likely it would become the last owner...

PPGe - the DisplaySaveDataInfo1 crash is new? I guess it must be related to the text drawer, maybe it's some save list with 100 saves and we ran out of space? Hm.

-[Unknown]

@hrydgard
Copy link
Owner Author

Yeah, that one's new. Far less common than the Download one.

@hrydgard
Copy link
Owner Author

Here's one that's not new, but it's pretty high in the chart:

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> org.ppsspp.ppsspp <<<

backtrace:
  #00  pc 0000000000499108  /data/app/org.ppsspp.ppsspp-6uhzoIIKLtxYP2LX_cVf7A==/split_config.arm64_v8a.apk!libppsspp_jni.so (offset 0x1000) (AfterMatchingMipsCall::run(MipsCall&)+80)
  #01  pc 00000000003fd494  /data/app/org.ppsspp.ppsspp-6uhzoIIKLtxYP2LX_cVf7A==/split_config.arm64_v8a.apk!libppsspp_jni.so (offset 0x1000) (HLEReturnFromMipsCall()+540)
  #02  pc 00000000003fd978  /data/app/org.ppsspp.ppsspp-6uhzoIIKLtxYP2LX_cVf7A==/split_config.arm64_v8a.apk!libppsspp_jni.so (offset 0x1000) (CallSyscallWithoutFlags(HLEFunction const*)+32)
  #03  pc 0000000000070740  <anonymous>

That's in the Adhoc matchmaking code it seems.

@hrydgard
Copy link
Owner Author

Here's another interesting one, pretty rare though:

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> org.ppsspp.ppsspp <<<

backtrace:
  #00  pc 00000000004c9662  /data/app/org.ppsspp.ppsspp-dHsZvlMFDlK8Cv_0xwbyJQ==/lib/arm/libppsspp_jni.so (GameScreen::render()+1529)
  #01  pc 00000000005555c9  /data/app/org.ppsspp.ppsspp-dHsZvlMFDlK8Cv_0xwbyJQ==/lib/arm/libppsspp_jni.so (ScreenManager::render()+84)
  #02  pc 00000000004a158f  /data/app/org.ppsspp.ppsspp-dHsZvlMFDlK8Cv_0xwbyJQ==/lib/arm/libppsspp_jni.so (NativeRender(GraphicsContext*)+502)
  #03  pc 000000000049b86f  /data/app/org.ppsspp.ppsspp-dHsZvlMFDlK8Cv_0xwbyJQ==/lib/arm/libppsspp_jni.so (UpdateRunLoopAndroid(_JNIEnv*)+22)
  #04  pc 000000000049d0d9  /data/app/org.ppsspp.ppsspp-dHsZvlMFDlK8Cv_0xwbyJQ==/lib/arm/libppsspp_jni.so
  #05  pc 00000000002ac665  /data/app/org.ppsspp.ppsspp-dHsZvlMFDlK8Cv_0xwbyJQ==/lib/arm/libppsspp_jni.so (void* std::__ndk1::__thread_proxy<std::__ndk1::tuple<std::__ndk1::unique_ptr<std::__ndk1::__thread_struct, std::__ndk1::default_delete<std::__ndk1::__thread_struct> >, void (*)()> >(void*)+24)
  #06  pc 0000000000047d3f  /system/lib/libc.so (__pthread_start(void*)+22)
  #07  pc 000000000001b029  /system/lib/libc.so (__start_thread+32)

@hrydgard
Copy link
Owner Author

And an old one that seems suspicious, though can also be a driver bug. Don't completely trust our descriptor set handling code....

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> org.ppsspp.ppsspp <<<

backtrace:
  #00  pc 00000000000cd6c0  /vendor/lib64/hw/vulkan.sdm660.so (A5xDescriptorSet::HwUpdateWriteDescriptors(QglDescriptorWriteData const*)+576)
  #01  pc 0000000000054d1c  /vendor/lib64/hw/vulkan.sdm660.so (qglinternal::vkUpdateDescriptorSets(VkDevice_T*, unsigned int, VkWriteDescriptorSet const*, unsigned int, VkCopyDescriptorSet const*)+532)
  #02  pc 000000000053e994  /data/app/org.ppsspp.ppsspp-71vIo6mZAROOgsixUKgVrw==/split_config.arm64_v8a.apk (offset 0x1000) (DrawEngineVulkan::GetOrCreateDescriptorSet(VkImageView_T*, VkSampler_T*, VkBuffer_T*, VkBuffer_T*, VkBuffer_T*, bool)+1308)
  #03  pc 000000000053f950  /data/app/org.ppsspp.ppsspp-71vIo6mZAROOgsixUKgVrw==/split_config.arm64_v8a.apk (offset 0x1000) (DrawEngineVulkan::DoFlush()+3184)
  #04  pc 00000000005b23e0  /data/app/org.ppsspp.ppsspp-71vIo6mZAROOgsixUKgVrw==/split_config.arm64_v8a.apk (offset 0x1000) (GPUCommon::FastRunLoop(DisplayList&)+192)
  #05  pc 00000000005b1f40  /data/app/org.ppsspp.ppsspp-71vIo6mZAROOgsixUKgVrw==/split_config.arm64_v8a.apk (offset 0x1000) (GPUCommon::InterpretList(DisplayList&)+904)
  #06  pc 00000000005b11a0  /data/app/org.ppsspp.ppsspp-71vIo6mZAROOgsixUKgVrw==/split_config.arm64_v8a.apk (offset 0x1000) (GPUCommon::ProcessDLQueue()+112)
  #07  pc 00000000005b1018  /data/app/org.ppsspp.ppsspp-71vIo6mZAROOgsixUKgVrw==/split_config.arm64_v8a.apk (offset 0x1000) (GPUCommon::EnqueueList(unsigned int, unsigned int, int, PSPPointer<PspGeListArgs>, bool)+1588)
  #08  pc 000000000041fb60  /data/app/org.ppsspp.ppsspp-71vIo6mZAROOgsixUKgVrw==/split_config.arm64_v8a.apk (offset 0x1000) (void WrapU_UUIU<&(sceGeListEnQueue(unsigned int, unsigned int, int, unsigned int))>()+60)
  #09  pc 00000000003fd978  /data/app/org.ppsspp.ppsspp-71vIo6mZAROOgsixUKgVrw==/split_config.arm64_v8a.apk (offset 0x1000) (CallSyscallWithoutFlags(HLEFunction const*)+32)
  #10  pc 0000000000022e28  <anonymous>

Anyway, should already be at an acceptable level, will probably release 1.10.1 tomorrow or the day after.

@hrydgard
Copy link
Owner Author

hrydgard commented Jul 3, 2020

Did a quiet 10% rollout of 1.10.1 on Android. Here's one I hadn't spotted before, 19 times on a single device "A44 (itel-A44)" (so probably not much to worry about, but got to be annoying for that person):

ktrace:
  #00  pc 000000000004ad34  /system/lib/libc.so (tgkill+12)
  #01  pc 00000000000484d3  /system/lib/libc.so (pthread_kill+34)
  #02  pc 000000000001d485  /system/lib/libc.so (raise+10)
  #03  pc 0000000000018fc1  /system/lib/libc.so (__libc_android_abort+34)
  #04  pc 0000000000017024  /system/lib/libc.so (abort+4)
  #05  pc 0000000000319845  /system/lib/libart.so (art::Runtime::Abort()+252)
  #06  pc 00000000000b4cb9  /system/lib/libart.so (art::LogMessage::~LogMessage()+864)
  #07  pc 0000000000332dd3  /system/lib/libart.so (art::Thread::AssertNoPendingException() const+602)
  #08  pc 00000000000e494d  /system/lib/libart.so (art::ClassLinker::FindClass(art::Thread*, char const*, art::Handle<art::mirror::ClassLoader>)+28)
  #09  pc 0000000000260ccf  /system/lib/libart.so (art::JNI::FindClass(_JNIEnv*, char const*)+1878)
  #10  pc 000000000049c603  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (__cameraGetDeviceList()+218)
  #11  pc 0000000000386329  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (Camera::getDeviceList()+4)
  #12  pc 00000000004ce08f  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (GameSettingsScreen::CreateViews()+14990)
  #13  pc 00000000005570e1  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UIScreen::DoRecreateViews()+140)
  #14  pc 000000000055731d  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UIScreen::render()+24)
  #15  pc 0000000000555539  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (ScreenManager::render()+84)
  #16  pc 00000000004a163f  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (NativeRender(GraphicsContext*)+502)
  #17  pc 000000000049b91f  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UpdateRunLoopAndroid(_JNIEnv*)+22)
  #18  pc 000000000049d189  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so
  #19  pc 00000000002ac6a5  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (void* std::__ndk1::__thread_proxy<std::__ndk1::tuple<std::__ndk1::unique_ptr<std::__ndk1::__thread_struct, std::__ndk1::default_delete<std::__ndk1::__thread_struct> >, void (*)()> >(void*)+24)
  #20  pc 0000000000047fa3  /system/lib/libc.so (__pthread_start(void*)+22)
  #21  pc 0000000000019a0d  /system/lib/libc.so (__start_thread+6)

Download crashes are gone.

This oldie but goodie is rising through the ranks:

  #00  pc 000000000001e698  /system/lib64/libc.so (abort+120)
  #01  pc 00000000000083e4  /system/lib64/liblog.so (__android_log_assert+296)
  #02  pc 0000000000618fac  /data/app/org.ppsspp.ppsspp-LoZRddpxiOkiOn5A8galeA==/lib/arm64/libppsspp_jni.so (AndroidAssertLog(char const*, char const*, int, char const*, char const*, ...)+168)
  #03  pc 0000000000627690  /data/app/org.ppsspp.ppsspp-LoZRddpxiOkiOn5A8galeA==/lib/arm64/libppsspp_jni.so (NativeInitGraphics(GraphicsContext*)+324)
  #04  pc 0000000000622a74  /data/app/org.ppsspp.ppsspp-LoZRddpxiOkiOn5A8galeA==/lib/arm64/libppsspp_jni.so
  #05  pc 0000000000395d50  /data/app/org.ppsspp.ppsspp-LoZRddpxiOkiOn5A8galeA==/lib/arm64/libppsspp_jni.so (void* std::__ndk1::__thread_proxy<std::__ndk1::tuple<std::__ndk1::unique_ptr<std::__ndk1::__thread_struct, std::__ndk1::default_delete<std::__ndk1::__thread_struct> >, void (*)()> >(void*)+44)
  #06  pc 0000000000077e08  /system/lib64/libc.so (__pthread_start(void*)+36)
  #07  pc 000000000001fa74  /system/lib64/libc.so (__start_thread+68)

This guy is still around:

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> org.ppsspp.ppsspp <<<

backtrace:
  #00  pc 000000000049a260  /data/app/org.ppsspp.ppsspp-TJeQtrHHAei0silzKB4lvg==/split_config.arm64_v8a.apk!libppsspp_jni.so (offset 0x1000) (AfterMatchingMipsCall::run(MipsCall&)+80)
  #01  pc 00000000003fd5d0  /data/app/org.ppsspp.ppsspp-TJeQtrHHAei0silzKB4lvg==/split_config.arm64_v8a.apk!libppsspp_jni.so (offset 0x1000) (HLEReturnFromMipsCall()+540)
  #02  pc 00000000003fdab4  /data/app/org.ppsspp.ppsspp-TJeQtrHHAei0silzKB4lvg==/split_config.arm64_v8a.apk!libppsspp_jni.so (offset 0x1000) (CallSyscallWithoutFlags(HLEFunction const*)+32)
  #03  pc 00000000000a8ce4  <anonymous>

Another new one I spotted while going through stuff that was buried before:

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> org.ppsspp.ppsspp <<<

backtrace:
  #00  pc 00000000000220ac  /system/lib64/libc.so (abort+116)
  #01  pc 0000000000c1de50  /data/app/org.ppsspp.ppsspp-aOgeuW2_Wvrb_KwVfJywhQ==/split_config.arm64_v8a.apk (offset 0x1000)
  #02  pc 0000000000c1ded0  /data/app/org.ppsspp.ppsspp-aOgeuW2_Wvrb_KwVfJywhQ==/split_config.arm64_v8a.apk (offset 0x1000)
  #03  pc 0000000000c1ae98  /data/app/org.ppsspp.ppsspp-aOgeuW2_Wvrb_KwVfJywhQ==/split_config.arm64_v8a.apk (offset 0x1000)
  #04  pc 0000000000c1ae40  /data/app/org.ppsspp.ppsspp-aOgeuW2_Wvrb_KwVfJywhQ==/split_config.arm64_v8a.apk (offset 0x1000) (std::terminate()+52)
  #05  pc 000000000048f950  /data/app/org.ppsspp.ppsspp-aOgeuW2_Wvrb_KwVfJywhQ==/split_config.arm64_v8a.apk (offset 0x1000) (__NetAdhocInit()+452)
  #06  pc 00000000004399f0  /data/app/org.ppsspp.ppsspp-aOgeuW2_Wvrb_KwVfJywhQ==/split_config.arm64_v8a.apk (offset 0x1000) (__KernelInit()+188)
  #07  pc 0000000000458b30  /data/app/org.ppsspp.ppsspp-aOgeuW2_Wvrb_KwVfJywhQ==/split_config.arm64_v8a.apk (offset 0x1000) (__KernelLoadExec(char const*, unsigned int, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>>*)+392)
  #08  pc 00000000004fa574  /data/app/org.ppsspp.ppsspp-aOgeuW2_Wvrb_KwVfJywhQ==/split_config.arm64_v8a.apk (offset 0x1000)
  #09  pc 0000000000091fac  /system/lib64/libc.so (__pthread_start(void*)+36)
  #10  pc 0000000000023968  /system/lib64/libc.so (__start_thread+68)

And a real oddball that's also not new:

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> org.ppsspp.ppsspp <<<

backtrace:
  #00  pc 0000000000558a1a  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UI::PopupMultiChoice::ChoiceCallback(int)+37)
  #01  pc 0000000000558561  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (ListPopupScreen::OnListChoice(UI::EventParams&)+48)
  #02  pc 000000000055e303  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UI::Event::Dispatch(UI::EventParams&)+30)
  #03  pc 0000000000552e65  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UI::DispatchEvents()+340)
  #04  pc 0000000000553ff1  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UI::UpdateViewHierarchy(UI::ViewGroup*)+1072)
  #05  pc 00000000005578e1  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (PopupScreen::update()+24)
  #06  pc 000000000055501d  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (ScreenManager::update()+44)
  #07  pc 00000000004a2067  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (NativeUpdate()+190)
  #08  pc 000000000049b913  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UpdateRunLoopAndroid(_JNIEnv*)+10)
  #09  pc 000000000049d189  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so
  #10  pc 00000000002ac6a5  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (void* std::__ndk1::__thread_proxy<std::__ndk1::tuple<std::__ndk1::unique_ptr<std::__ndk1::__thread_struct, std::__ndk1::default_delete<std::__ndk1::__thread_struct> >, void (*)()> >(void*)+24)
  #11  pc 0000000000041adb  /system/lib/libc.so (__pthread_start(void*)+30)
  #12  pc 0000000000019355  /system/lib/libc.so (__start_thread+6)

@hrydgard
Copy link
Owner Author

hrydgard commented Jul 3, 2020

Here's an ANR that might be slightly serious, seems to be from using joypad on android while downloads are cancelling.

one thread:

  #00  pc 000000000001ed2c  /system/lib64/libc.so (syscall+28)
  #01  pc 000000000002202c  /system/lib64/libc.so (__futex_wait_ex(void volatile*, bool, int, bool, timespec const*)+140)
  #02  pc 00000000000840d0  /system/lib64/libc.so (NonPI::MutexLockWithTimeout(pthread_mutex_internal_t*, bool, timespec const*)+632)
  #03  pc 0000000000c10e88  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (std::__ndk1::recursive_mutex::lock()+8)
  #04  pc 000000000071be8c  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (ScreenManager::axis(AxisInput const&)+36)
  #05  pc 000000000062996c  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (NativeAxis(AxisInput const&)+584)
  #06  pc 0000000000620d30  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (Java_org_ppsspp_ppsspp_NativeApp_accelerometer+84)
  at org.ppsspp.ppsspp.NativeApp.accelerometer (Native method)
  at org.ppsspp.ppsspp.NativeGLView.onSensorChanged (NativeGLView.java:114)
  at android.hardware.SystemSensorManager$SensorEventQueue.dispatchSensorEvent (SystemSensorManager.java:833)
  at android.os.MessageQueue.nativePollOnce (Native method)
  at android.os.MessageQueue.next (MessageQueue.java:327)
  at android.os.Looper.loop (Looper.java:196)
  at android.app.ActivityThread.main (ActivityThread.java:6760)
  at java.lang.reflect.Method.invoke (Native method)
  at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run (RuntimeInit.java:493)
  at com.android.internal.os.ZygoteInit.main (ZygoteInit.java:911)

Another thread:

  #00  pc 000000000001ed2c  /system/lib64/libc.so (syscall+28)
  #01  pc 0000000000083720  /system/lib64/libc.so (pthread_join+208)
  #02  pc 0000000000c189fc  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (std::__ndk1::thread::join()+28)
  #03  pc 00000000006ef228  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (http::Downloader::CancelAll()+136)
  #04  pc 00000000006b325c  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (StoreScreen::~StoreScreen()+44)
  #05  pc 00000000006b33ec  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (StoreScreen::~StoreScreen()+16)
  #06  pc 000000000071c2e0  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (ScreenManager::processFinishDialog()+368)
  #07  pc 00000000006287a8  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (NativeRender(GraphicsContext*)+416)
  #08  pc 00000000006207dc  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (UpdateRunLoopAndroid(_JNIEnv*)+48)
  #09  pc 0000000000622ab4  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (???)
  #10  pc 0000000000395d50  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (void* std::__ndk1::__thread_proxy<std::__ndk1::tuple<std::__ndk1::unique_ptr<std::__ndk1::__thread_struct, std::__ndk1::default_delete<std::__ndk1::__thread_struct> >, void (*)()> >(void*)+44)
  #11  pc 0000000000083114  /system/lib64/libc.so (__pthread_start(void*)+36)
  #12  pc 00000000000233bc  /system/lib64/libc.so (__start_thread+68)

Third thread:

  #00  pc 000000000001ed2c  /system/lib64/libc.so (syscall+28)
  #01  pc 000000000002202c  /system/lib64/libc.so (__futex_wait_ex(void volatile*, bool, int, bool, timespec const*)+140)
  #02  pc 000000000008261c  /system/lib64/libc.so (pthread_cond_wait+60)
  #03  pc 0000000000bd7910  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (std::__ndk1::condition_variable::wait(std::__ndk1::unique_lock<std::__ndk1::mutex>&)+20)
  #04  pc 00000000006fdd50  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (GLRenderManager::ThreadFrame()+196)
  at org.ppsspp.ppsspp.NativeRenderer.displayRender (Native method)
  at org.ppsspp.ppsspp.NativeRenderer.onDrawFrame (NativeRenderer.java:27)
  at android.opengl.GLSurfaceView$GLThread.guardedRun (GLSurfaceView.java:1578)
  at android.opengl.GLSurfaceView$GLThread.run (GLSurfaceView.java:1274)

Downloader::Do:

"Downloader::Do" prio=5 (not attached)
  | sysTid=21245 nice=-10 cgrp=default
  | state=S schedstat=( 4981001 4416999 9 ) utm=0 stm=0 core=4 HZ=100
  #00  pc 0000000000070164  /system/lib64/libc.so (recvfrom+4)
  #01  pc 00000000006c2ab4  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (Buffer::Read(int, unsigned long)+124)
  #02  pc 00000000006ec9e8  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (http::Client::ReadResponseHeaders(Buffer*, std::__ndk1::vector<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>>, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>>>>&, float*)+96)
  #03  pc 00000000006ec860  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (http::Client::GET(char const*, Buffer*, std::__ndk1::vector<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>>, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>>>>&, float*, bool*)+144)
  #04  pc 00000000006ee508  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (http::Download::PerformGET(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>> const&)+968)
  #05  pc 00000000006edbd8  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (http::Download::Do()+284)
  #06  pc 00000000006ef598  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (void* std::__ndk1::__thread_proxy<std::__ndk1::tuple<std::__ndk1::unique_ptr<std::__ndk1::__thread_struct, std::__ndk1::default_delete<std::__ndk1::__thread_struct> >, std::__ndk1::__bind<void (http::Download::*)(), http::Download*> > >(void*)+64)
  #07  pc 0000000000083114  /system/lib64/libc.so (__pthread_start(void*)+36)
  #08  pc 00000000000233bc  /system/lib64/libc.so (__start_thread+68)

Not all of these are necessarily involved in the likely deadlock though.


  #00  pc 000000000001ed2c  /system/lib64/libc.so (syscall+28)
  #01  pc 000000000002202c  /system/lib64/libc.so (__futex_wait_ex(void volatile*, bool, int, bool, timespec const*)+140)
  #02  pc 000000000008261c  /system/lib64/libc.so (pthread_cond_wait+60)
  #03  pc 0000000000bd7910  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (std::__ndk1::condition_variable::wait(std::__ndk1::unique_lock<std::__ndk1::mutex>&)+20)
  #04  pc 0000000000717248  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (PrioritizedWorkQueue::Pop()+156)
  #05  pc 00000000007175ec  /data/app/org.ppsspp.ppsspp-oR8sD6RmGsBMttZpgzZfXw==/split_config.arm64_v8a.apk (???)
  #06  pc 0000000000083114  /system/lib64/libc.so (__pthread_start(void*)+36)
  #07  pc 00000000000233bc  /system/lib64/libc.so (__start_thread+68)

@hrydgard
Copy link
Owner Author

hrydgard commented Jul 3, 2020

Hm, I think it's possible that the above one is simply a Buffer::Read blocking for a long time, making cancellation kinda not happen.

@unknownbrackets
Copy link
Collaborator

Agreed, that's what it looks like. We'd need to convert to non-blocking IO or send a signal to cancel.

-[Unknown]

@hrydgard
Copy link
Owner Author

hrydgard commented Jul 4, 2020

InstallZipScreen???

backtrace:
  #00  pc 00000000006bca30  /data/app/org.ppsspp.ppsspp-1/lib/arm64/libppsspp_jni.so (InstallZipScreen::update()+192)
  #01  pc 000000000071b92c  /data/app/org.ppsspp.ppsspp-1/lib/arm64/libppsspp_jni.so (ScreenManager::update()+72)
  #02  pc 0000000000629400  /data/app/org.ppsspp.ppsspp-1/lib/arm64/libppsspp_jni.so (NativeUpdate()+276)
  #03  pc 00000000006207cc  /data/app/org.ppsspp.ppsspp-1/lib/arm64/libppsspp_jni.so (UpdateRunLoopAndroid(_JNIEnv*)+32)
  #04  pc 0000000000622ab4  /data/app/org.ppsspp.ppsspp-1/lib/arm64/libppsspp_jni.so
  #05  pc 0000000000395d50  /data/app/org.ppsspp.ppsspp-1/lib/arm64/libppsspp_jni.so (void* std::__ndk1::__thread_proxy<std::__ndk1::tuple<std::__ndk1::unique_ptr<std::__ndk1::__thread_struct, std::__ndk1::default_delete<std::__ndk1::__thread_struct> >, void (*)()> >(void*)+44)
  #06  pc 0000000000067508  /system/lib64/libc.so (__pthread_start(void*)+52)
  #07  pc 000000000001ed84  /system/lib64/libc.so (__start_thread+16)

@hrydgard
Copy link
Owner Author

hrydgard commented Jul 5, 2020

Found another oddity, I think things have gone pretty wrong already when we get something like this:

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> org.ppsspp.ppsspp <<<

backtrace:
  #00  pc 00000000003afe76  /data/app/org.ppsspp.ppsspp-Msn-tDGoUw-6IPEZ50vqRw==/split_config.armeabi_v7a.apk (offset 0x1000) (Memory::Write_Opcode_JIT(unsigned int, Memory::Opcode const&)+14)
  #01  pc 000000000039b2a1  /data/app/org.ppsspp.ppsspp-Msn-tDGoUw-6IPEZ50vqRw==/split_config.armeabi_v7a.apk (offset 0x1000) (JitBlockCache::FinalizeBlock(int, bool)+76)
  #02  pc 000000000028ddd3  /data/app/org.ppsspp.ppsspp-Msn-tDGoUw-6IPEZ50vqRw==/split_config.armeabi_v7a.apk (offset 0x1000) (MIPSComp::ArmJit::Compile(unsigned int)+118)
  #03  pc 0000000000000106  <anonymous>

hrydgard added a commit that referenced this issue Jul 6, 2020
@hrydgard
Copy link
Owner Author

hrydgard commented Jul 7, 2020

Here's an interesting one that I got a single one of:

cktrace:
  #00  pc 0000000000606e64  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (VulkanDeviceAllocator::Free(VkDeviceMemory_T*, unsigned long)+140)
  #01  pc 0000000000602360  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (VulkanTexture::Destroy()+160)
  #02  pc 000000000070c964  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (Draw::VKTexture::~VKTexture()+40)
  #03  pc 00000000006f9a28  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (Draw::RefCountedObject::Release()+56)
  #04  pc 00000000006b86d0  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (HttpImageFileView::~HttpImageFileView()+68)
  #05  pc 00000000006b8784  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (HttpImageFileView::~HttpImageFileView()+16)
  #06  pc 0000000000731900  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (UI::ViewGroup::Clear()+100)
  #07  pc 00000000006b363c  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (ProductView::CreateViews()+56)
  #08  pc 00000000006b50e4  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (ProductView::Update()+44)
  #09  pc 0000000000734794  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (UI::ScrollView::Update()+92)
  #10  pc 00000000007321a0  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (UI::ViewGroup::Update()+80)
  #11  pc 00000000007321a0  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (UI::ViewGroup::Update()+80)
  #12  pc 000000000071c848  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (UI::UpdateViewHierarchy(UI::ViewGroup*)+1456)
  #13  pc 00000000006b582c  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (StoreScreen::update()+40)
  #14  pc 000000000071de1c  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (ScreenManager::update()+72)
  #15  pc 00000000006294ac  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (NativeUpdate()+276)
  #16  pc 0000000000622360  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/split_config.arm64_v8a.apk (offset 0x1000) (Java_org_ppsspp_ppsspp_NativeActivity_runEGLRenderLoop+632)
  #17  pc 00000000000031b8  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/oat/arm64/base.odex (offset 0x3000) (org.ppsspp.ppsspp.NativeActivity.runEGLRenderLoop+152)
  #18  pc 000000000055e988  /system/lib64/libart.so (art_quick_invoke_stub+584)
  #19  pc 00000000000cf940  /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200)
  #20  pc 0000000000283870  /system/lib64/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+344)
  #21  pc 000000000027d82c  /system/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+948)
  #22  pc 0000000000531058  /system/lib64/libart.so (MterpInvokeVirtualQuick+584)
  #23  pc 0000000000554c94  /system/lib64/libart.so (ExecuteMterpImpl+29972)
  #24  pc 0000000000008efc  /data/app/org.ppsspp.ppsspp-Fy20XW7zBZ-1PYbvBBYzQQ==/oat/arm64/base.vdex (org.ppsspp.ppsspp.NativeActivity$1.run+72)
  #25  pc 0000000000257328  /system/lib64/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool) (.llvm.1635852419)+496)
  #26  pc 000000000051e748  /system/lib64/libart.so (artQuickToInterpreterBridge+1032)
  #27  pc 0000000000567afc  /system/lib64/libart.so (art_quick_to_interpreter_bridge+92)
  #28  pc 000000000025d2e8  /system/framework/arm64/boot.oat (offset 0x114000) (java.lang.Thread.run+72)
  #29  pc 000000000055e988  /system/lib64/libart.so (art_quick_invoke_stub+584)
  #30  pc 00000000000cf940  /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200)
  #31  pc 0000000000464968  /system/lib64/libart.so (art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*)+104)
  #32  pc 0000000000465a30  /system/lib64/libart.so (art::InvokeVirtualOrInterfaceWithJValues(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, jvalue*)+424)
  #33  pc 0000000000490da8  /system/lib64/libart.so (art::Thread::CreateCallback(void*)+1120)
  #34  pc 0000000000083814  /system/lib64/libc.so (__pthread_start(void*)+36)
  #35  pc 000000000002340c  /system/lib64/libc.so (__start_thread+68)

@unknownbrackets
Copy link
Collaborator

Closing this for #14082.

-[Unknown]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants