Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The 1.14 Google Play mystery crash thread #16601

Closed
hrydgard opened this issue Dec 16, 2022 · 25 comments
Closed

The 1.14 Google Play mystery crash thread #16601

hrydgard opened this issue Dec 16, 2022 · 25 comments
Milestone
v1.15.0

Comments

@hrydgard
Copy link
Owner

hrydgard commented Dec 16, 2022 •
edited

More than five years after I first requested it, the Google Play gods have finally answered my prayers and now display assert messages along with stack traces:

image

This means that we can now pass along information about which game was running when we hit an assert, and similar things!

Unfortunately they also made the stack trace UI incredibly slow. Oh well.

There are a whole bunch of OpenGL initialization crashes as usual, now collected better thanks to some asserts I added like that one, but I still think should probably avoid crashing, show a message and bail, or even automatically restart the activity. The few times I've been able to randomly repro these (almost entirely on very old devices), restarting the app helped.

Anyway, if they fix their server issues, I'll add some more mysterious ones here. I did see one in UninstallExceptionHandler but now I can't get it to display it again.

Hm, seems the UninstallExceptionHandler one is just some somewhat worrying memory corruption:

 #00  pc 0x000000000004e178  /apex/com.android.runtime/lib64/bionic/libc.so (abort+168)
  #01  pc 0x0000000000042818  /apex/com.android.runtime/lib64/bionic/libc.so (scudo::die()+8)
  #02  pc 0x0000000000042e50  /apex/com.android.runtime/lib64/bionic/libc.so (scudo::ScopedErrorReport::~ScopedErrorReport()+32)
  #03  pc 0x0000000000042f18  /apex/com.android.runtime/lib64/bionic/libc.so (scudo::reportHeaderCorruption(void*)+60)
  #04  pc 0x00000000000444a0  /apex/com.android.runtime/lib64/bionic/libc.so (scudo::Allocator<scudo::AndroidConfig, &(scudo_malloc_postinit)>::deallocate(void*, scudo::Chunk::Origin, unsigned long, unsigned long)+332)
  #05  pc 0x00000000007646e8  /data/app/~~_MCmDAcVR2sC4CsnTSQpyg==/org.ppsspp.ppsspp-xYIsGg1ImySc_IODjV9WhQ==/split_config.arm64_v8a.apk!libppsspp_jni.so (UninstallExceptionHandler()+116)
  #06  pc 0x00000000005d968c  /data/app/~~_MCmDAcVR2sC4CsnTSQpyg==/org.ppsspp.ppsspp-xYIsGg1ImySc_IODjV9WhQ==/split_config.arm64_v8a.apk!libppsspp_jni.so (CPU_Shutdown()+16)

This surely must be some kind of shutdown bug? Unless there's some curRenderStep_ confusion (== 0 for some reason)...

backtrace:
  #00  pc 0x000000000049587e  _v7a.apk!libppsspp_jni.so (GLRenderManager::SetNoBlendAndMask(int)+106)
  #01  pc 0x0000000000495311  _v7a.apk!libppsspp_jni.so (DrawEngineGLES::ApplyDrawState(int)+288)
  #02  pc 0x0000000000499071  _v7a.apk!libppsspp_jni.so (DrawEngineGLES::DoFlush()+1664)
  #03  pc 0x00000000005087f9  _v7a.apk!libppsspp_jni.so (GPUCommon::FastRunLoop(DisplayList&)+192)
  #04  pc 0x000000000050838f  _v7a.apk!libppsspp_jni.so (GPUCommon::InterpretList(DisplayList&)+410)
  #05  pc 0x0000000000507c4d  _v7a.apk!libppsspp_jni.so (GPUCommon::ProcessDLQueue()+104)
  #06  pc 0x0000000000507b13  _v7a.apk!libppsspp_jni.so (GPUCommon::EnqueueList(unsigned int, unsigned int, int, PSPPointer<PspGeListArgs>, bool)+1462)
  #07  pc 0x00000000003a1519  _v7a.apk!libppsspp_jni.so (void WrapU_UUIU<&(sceGeListEnQueue(unsigned int, unsigned int, int, unsigned int))>()+48)
  #08  pc 0x00000000003842a5  _v7a.apk!libppsspp_jni.so (CallSyscallWithoutFlags(HLEFunction const*)+32)
  #09  pc 0x0000000000002022  [heap]

Locking a destroyed mutex, seems like in a VK driver:

  #00  pc 0x0000000000078b9c  /apex/com.android.runtime/lib64/bionic/libc.so (abort+180)
  #01  pc 0x000000000007a9fc  /apex/com.android.runtime/lib64/bionic/libc.so (__fortify_fatal(char const*, ...)+124)
  #02  pc 0x00000000000dd1cc  /apex/com.android.runtime/lib64/bionic/libc.so (HandleUsingDestroyedMutex(pthread_mutex_t*, char const*)+60)
  #03  pc 0x00000000000dd00c  /apex/com.android.runtime/lib64/bionic/libc.so (pthread_mutex_lock+172)
  #04  pc 0x00000000006cf7e8  /vendor/lib64/egl/libGLES_mali.so
  #05  pc 0x00000000007eba54  /vendor/lib64/egl/libGLES_mali.so
  #06  pc 0x00000000006cd898  /vendor/lib64/egl/libGLES_mali.so
  #07  pc 0x00000000006cc328  /vendor/lib64/egl/libGLES_mali.so
  #08  pc 0x00000000006cb320  /data/app/~~Z48YEGq3eSqmDamiLrUOWg==/org.ppsspp.ppsspp-uprapXiBl8CpmMwHp2grWA==/split_config.arm64_v8a.apk!libppsspp_jni.so (VKRGraphicsPipeline::Create(VulkanContext*)+80)
  #09  pc 0x00000000006cce4c  /data/app/~~Z48YEGq3eSqmDamiLrUOWg==/org.ppsspp.ppsspp-uprapXiBl8CpmMwHp2grWA==/split_config.arm64_v8a.apk!libppsspp_jni.so (VulkanRenderManager::CompileThreadFunc()+256)
  #10  pc 0x00000000006d0e2c  /data/app/~~Z48YEGq3eSqmDamiLrUOWg==/org.ppsspp.ppsspp-uprapXiBl8CpmMwHp2grWA==/split_config.arm64_v8a.apk!libppsspp_jni.so (void* std::__ndk1::__thread_proxy<std::__ndk1::tuple<std::__ndk1::unique_ptr<std::__ndk1::__thread_struct, std::__ndk1::default_delete<std::__ndk1::__thread_struct> >, void (VulkanRenderManager::*)(), VulkanRenderManager*> >(void*)+64)
@hrydgard hrydgard added this to the v1.14.1 milestone Dec 16, 2022
@hrydgard hrydgard mentioned this issue Dec 16, 2022
Merged
@unknownbrackets
Copy link
Collaborator

Hm, SetNoBlendAndMask would be the first opportunity for ApplyDrawState() to interact with a missing curRenderStep_ in a lot of cases.

(VKRGraphicsPipeline::Create(VulkanContext*)+80) seems relatively early for the Vk calls in that func, but... if this isn't a driver bug it seems like it would either need to be a shutdown hazard or a pipeline cache lifetime issue, right?

-[Unknown]

@hrydgard hrydgard modified the milestones: v1.14.1, v1.14.2 Dec 20, 2022
@hrydgard
Copy link
Owner Author

hrydgard commented Dec 20, 2022 •
edited

Found a hang reported, where one of the threads were sitting here, very odd:

#00  pc 0x000000000055c5e0  !libppsspp_jni.so (JitBlockCache::GetBlockNumbersFromAddress(unsigned int, std::__ndk1::vector<int, std::__ndk1::allocator<int> >*)+128)
#01  pc 0x00000000003d5604 !libppsspp_jni.so (MIPSComp::Arm64Jit::DescribeCodePtr(unsigned char const*, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >&)+176)
#02  pc 0x0000000000574f20 !libppsspp_jni.so (Memory::HandleFault(unsigned long, void*)+300)

Wonder if we're just handling a fault over and over, or if this was just random..

@unknownbrackets unknownbrackets modified the milestones: v1.14.2, v1.14.3 Dec 30, 2022
@hrydgard
Copy link
Owner Author

hrydgard commented Dec 30, 2022 •
edited

Alright, so now after 1.14 there are some updates:

  • The very noisy GLES crashes are gone
  • Hangs are relatively getting more common (probably not a real change).
    • One hang I've seen a couple variants of is waiting for the vulkan render loop to exit, which doesn't happen for whatever reason then. Google Play reports this either as stuck in Java_org_ppsspp_ppsspp_NativeActivity_requestExitVulkanRenderLoop (which ends with a simple wait loop) or stuck in the actual thread.

Strange TransformUnit::Flush crash (software renderer):

backtrace:
  #00  pc 0x00000000006c2070  libppsspp_jni.so (TransformUnit::Flush(char const*)+12)
  #01  pc 0x00000000006936d8  libppsspp_jni.so (GPUCommon::InterpretList(DisplayList&)+688)
  #02  pc 0x0000000000692bd0  libppsspp_jni.so (GPUCommon::ProcessDLQueue()+100)
  #03  pc 0x0000000000693068  libppsspp_jni.so (GPUCommon::UpdateStall(int, unsigned int)+88)
  #04  pc 0x00000000004ce294  libppsspp_jni.so
  #05  pc 0x00000000004a7058  libppsspp_jni.so (CallSyscallWithoutFlags(HLEFunction const*)+52)
  #06  pc 0x0000000000007820

Wacky bad_alloc in draw buffer (likely just OOM?):

Thread
terminating with uncaught exception of type std::bad_alloc: std::bad_alloc
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 30603 >>> org.ppsspp.ppsspp <<<

backtrace:
  #00  pc 0x00000000000895ec  /apex/com.android.runtime/lib64/bionic/libc.so (abort+180)
  #01  pc 0x0000000000e22960  libppsspp_jni.so
  #02  pc 0x0000000000e22ab8  libppsspp_jni.so
  #03  pc 0x0000000000e1f9a8  libppsspp_jni.so
  #04  pc 0x0000000000e1efd4  libppsspp_jni.so
  #05  pc 0x0000000000e1ef30  libppsspp_jni.so (__cxa_throw+112)
  #06  pc 0x0000000000e156c0  libppsspp_jni.so (operator new(unsigned long)+96)
  #07  pc 0x0000000000612a70  libppsspp_jni.so (void std::__ndk1::vector<VKRGraphicsPipeline*, std::__ndk1::allocator<VKRGraphicsPipeline*> >::__push_back_slow_path<VKRGraphicsPipeline* const&>(VKRGraphicsPipeline* const&)+120)
  #08  pc 0x0000000000c597a4  libppsspp_jni.so (Draw::VKContext::DrawUP(void const*, int)+452)
  #09  pc 0x0000000000770588  libppsspp_jni.so (DrawBuffer::Flush(bool)+132)
  #10  pc 0x0000000000765aa4  libppsspp_jni.so (UIContext::Flush()+28)
  #11  pc 0x000000000078c464  libppsspp_jni.so (EmuScreen::renderUI()+3404)
  #12  pc 0x000000000078b3b0  libppsspp_jni.so (EmuScreen::render()+216)
  #13  pc 0x0000000000c61050  libppsspp_jni.so (ScreenManager::render()+212)
  #14  pc 0x0000000000781a74  libppsspp_jni.so (NativeRender(GraphicsContext*)+696)
  #15  pc 0x000000000077a5e8  libppsspp_jni.so (Java_org_ppsspp_ppsspp_NativeActivity_runVulkanRenderLoop+428)

@hrydgard
Copy link
Owner Author

hrydgard commented Dec 30, 2022 •
edited

ReadFromZip:

  #00  pc 0x0000000000c83430  libppsspp_jni.so (_zip_error_set_from_source+32)
  #01  pc 0x0000000000c88014  libppsspp_jni.so (zip_source_open+208)
  #02  pc 0x0000000000c87fb0  libppsspp_jni.so (zip_source_open+108)
  #03  pc 0x0000000000c87fb0  libppsspp_jni.so (zip_source_open+108)
  #04  pc 0x0000000000c87fb0  libppsspp_jni.so (zip_source_open+108)
  #05  pc 0x0000000000c87fb0  libppsspp_jni.so (zip_source_open+108)
  #06  pc 0x0000000000c8440c  libppsspp_jni.so (zip_fopen_index_encrypted+56)
  #07  pc 0x0000000000c4bca4  libppsspp_jni.so (ReadFromZip(zip*, char const*, unsigned long*)+52)
  #08  pc 0x0000000000c4bfbc  libppsspp_jni.so (ZipAssetReader::ReadAsset(char const*, unsigned long*)+92)
  #09  pc 0x00000000007157c8  libppsspp_jni.so (VFSReadFile(char const*, unsigned long*)+324)
  #10  pc 0x00000000007917fc  libppsspp_jni.so
  #11  pc 0x0000000000790680  libppsspp_jni.so (GameInfoWorkItem::Run()+1792)
  #12  pc 0x0000000000763514  libppsspp_jni.so
  #13  pc 0x0000000000764da8  libppsspp_jni.so (void* std::__ndk1::__thread_proxy<std::__ndk1::tuple<std::__ndk1::unique_ptr<std::__ndk1::__thread_struct, std::__ndk1::default_delete<std::__ndk1::__thread_struct>>, void (*)(GlobalThreadContext*, ThreadContext*), GlobalThreadContext*, ThreadContext*>>(void*)+48)
  #14  pc 0x00000000000d4358  /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+36)
  #15  pc 0x0000000000071aa8  /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64)

Mic input (!)


  #00  pc 0x000000000004d4b4  /apex/com.android.runtime/lib64/bionic/libc.so (__memcpy+276)
  #01  pc 0x0000000000584490  libppsspp_jni.so (QueueBuf::pop(unsigned char*, unsigned int)+136)
  #02  pc 0x0000000000584948  libppsspp_jni.so (__MicInput(unsigned int, unsigned int, unsigned int, MICTYPE, bool)+408)
  #03  pc 0x000000000058217c  libppsspp_jni.so
  #04  pc 0x00000000004a7058  libppsspp_jni.so (CallSyscallWithoutFlags(HLEFunction const*)+52)
  #05  pc 0x000000000004b444

JIT finalize block:

backtrace:
  #00  pc 0x00000000003116b0  libppsspp_jni.so
  #01  pc 0x00000000003114c1  libppsspp_jni.so
  #02  pc 0x00000000003104fb  libppsspp_jni.so (JitBlockCache::FinalizeBlock(int, bool)+194)
  #03  pc 0x0000000000304c09  libppsspp_jni.so (MIPSComp::ArmJit::Compile(unsigned int)+140)
  #04  pc 0x0000000000000106

@unknownbrackets
Copy link
Collaborator

Strange TransformUnit::Flush crash (software renderer):

Hm, unless binner_ is null or broken or something, not sure. Memory corruption? OOM when even allocating binner? It is a bit large with all its buffers.

ReadFromZip:

Strange. Not sure if we end up in the "layered" case, but this is a read from assets even? That zip shouldn't be corrupt or anything... it should be a user zip.

Mic input (!)

__MicInput doesn't seem to check the pointer validity at all, but this seems to be to a newd temp buffer. Maybe even also OOM...

Not sure much about the others.

-[Unknown]

@unknownbrackets unknownbrackets mentioned this issue Dec 31, 2022
Merged
@hrydgard
Copy link
Owner Author

Been using addr2line on some stack traces that didn't resolve in the UI, there's one in the texture replacer here:

		ParallelRangeLoop(&g_threadManager, [&](int l, int h) {
			for (int y = l; y < h; ++y) {
				memcpy((uint8_t *)out + rowPitch * y, &data[0] + info.w * 4 * y, info.w * 4);  << bad
			}
		}, 0, info.h, MIN_LINES_PER_THREAD);

Also made a bunch of fixes in #16683

@hrydgard
Copy link
Owner Author

The most common hang has the following as the only interesting stack trace:

  #00  pc 0x0000000000086b8c  /apex/com.android.runtime/lib64/bionic/libc.so (syscall+28)
  #01  pc 0x000000000008a938  /apex/com.android.runtime/lib64/bionic/libc.so (__futex_wait_ex(void volatile*, bool, int, bool, timespec const*)+144)
  #02  pc 0x00000000000ec784  /apex/com.android.runtime/lib64/bionic/libc.so (NonPI::MutexLockWithTimeout(pthread_mutex_internal_t*, bool, timespec const*)+688)
  #03  pc 0x0000000000e15010  libppsspp_jni.so (std::__ndk1::recursive_mutex::lock()+8)
  #04  pc 0x0000000000c60cf8  libppsspp_jni.so (ScreenManager::axis(AxisInput const&)+60)
  #05  pc 0x0000000000782a60  libppsspp_jni.so (NativeAxis(AxisInput const&)+304)
  #06  pc 0x00000000007790cc  libppsspp_jni.so (Java_org_ppsspp_ppsspp_NativeApp_accelerometer+84)

This is the only other stack that had PPSSPP code in it:

  #00  pc 0x0000000000086b8c  /apex/com.android.runtime/lib64/bionic/libc.so (syscall+28)
  #01  pc 0x000000000008a938  /apex/com.android.runtime/lib64/bionic/libc.so (__futex_wait_ex(void volatile*, bool, int, bool, timespec const*)+144)
  #02  pc 0x00000000000eab90  /apex/com.android.runtime/lib64/bionic/libc.so (pthread_cond_wait+60)
  #03  pc 0x0000000000dda8e0  libppsspp_jni.so (std::__ndk1::condition_variable::wait(std::__ndk1::unique_lock<std::__ndk1::mutex>&)+20)
  #04  pc 0x0000000000763560  libppsspp_jni.so
  #05  pc 0x0000000000764da8  libppsspp_jni.so (void* std::__ndk1::__thread_proxy<std::__ndk1::tuple<std::__ndk1::unique_ptr<std::__ndk1::__thread_struct, std::__ndk1::default_delete<std::__ndk1::__thread_struct> >, void (*)(GlobalThreadContext*, ThreadContext*), GlobalThreadContext*, ThreadContext*> >(void*)+48)

which makes me think it's during shutdown or startup.

@unknownbrackets unknownbrackets mentioned this issue Dec 31, 2022
Merged
@hrydgard
Copy link
Owner Author

hrydgard commented Jan 1, 2023

The second most common hang is a Vulkan shutdown hang, with the following stack traces:

#02  pc 0x0000000000e1d4dc  libppsspp_jni.so (std::__ndk1::thread::join()+28)
#03  pc 0x00000000007442e4  libppsspp_jni.so (VulkanRenderManager::StopThread()+432)
#04  pc 0x00000000007444a8  libppsspp_jni.so (VulkanRenderManager::DestroyBackbuffers()+16)
#05  pc 0x000000000077c8e4  libppsspp_jni.so (AndroidVulkanContext::ShutdownFromRenderThread()+80)
#06  pc 0x000000000077a678  libppsspp_jni.so (Java_org_ppsspp_ppsspp_NativeActivity_runVulkanRenderLoop+572)

#00  pc 0x00000000000c67f8  /apex/com.android.runtime/lib64/bionic/libc.so (nanosleep+8)
#01  pc 0x000000000008b6bc  /apex/com.android.runtime/lib64/bionic/libc.so (usleep+76)
#02  pc 0x0000000000779870  !libppsspp_jni.so (Java_org_ppsspp_ppsspp_NativeActivity_requestExitVulkanRenderLoop+96)
at org.ppsspp.ppsspp.NativeActivity.requestExitVulkanRenderLoop (Native method)
at org.ppsspp.ppsspp.NativeActivity.joinRenderLoopThread (NativeActivity.java:718)
at org.ppsspp.ppsspp.NativeActivity.onPause (NativeActivity.java:805)
at android.app.Activity.performPause (Activity.java:8333)
at android.app.Instrumentation.callActivityOnPause (Instrumentation.java:1510)

@hrydgard
Copy link
Owner Author

hrydgard commented Jan 1, 2023

Another, I think, GL one:

#00  pc 0x000000000009a77c  /apex/com.android.runtime/lib/bionic/libc.so (getuid+12)
#01  pc 0x000000000000a41d  /system/lib/liblog.so (PmsgWrite(log_id, timespec*, iovec*, unsigned int)+320)
#02  pc 0x0000000000004ee9  /system/lib/liblog.so (write_to_log(log_id, iovec*, unsigned int)+128)
#03  pc 0x0000000000004e4f  /system/lib/liblog.so (__android_log_logd_logger+66)
#04  pc 0x000000000000c923  /system/lib/libbase.so (android::base::LogdLogChunk(android::base::LogId, android::base::LogSeverity, char const*, char const*)+98)
#05  pc 0x000000000000c87f  /system/lib/libbase.so (void android::base::SplitByLogdChunks<void (android::base::LogId, android::base::LogSeverity, char const*, char const*)>(android::base::LogId, android::base::LogSeverity, char const*, char const*, unsigned int, char const*, void  const(&)(android::base::LogId, android::base::LogSeverity, char const*, char const*))+834)
#06  pc 0x000000000000c535  /system/lib/libbase.so (android::base::LogdLogger::operator()(android::base::LogId, android::base::LogSeverity, char const*, char const*, unsigned int, char const*)+30)
#07  pc 0x000000000001a24d  /system/lib/libartbase.so (std::__1::__function::__func<art::InitLogging(char**, void (&)(char const*))::LogdLoggerLocked, std::__1::allocator<art::InitLogging(char**, void (&)(char const*))::LogdLoggerLocked>, void (android::base::LogId, android::base::LogSeverity, char const*, char const*, unsigned int, char const*)>::operator()(android::base::LogId&&, android::base::LogSeverity&&, char const*&&, char const*&&, unsigned int&&, char const*&&)+76)
#08  pc 0x000000000000d8a5  /system/lib/libbase.so (android::base::SetLogger(std::__1::function<void (android::base::LogId, android::base::LogSeverity, char const*, char const*, unsigned int, char const*)>&&)::$_2::__invoke(__android_log_message const*)+120)
#09  pc 0x00000000000050c9  /system/lib/liblog.so (__android_log_write_log_message+88)
#10  pc 0x0000000000005281  /system/lib/liblog.so (__android_log_print+112)
#11  pc 0x00000000005b6c73  !libppsspp_jni.so (AndroidLogger::Log(LogMessage const&)+82)
#12  pc 0x00000000005ad797  !libppsspp_jni.so (LogManager::Log(LogTypes::LOG_LEVELS, LogTypes::LOG_TYPE, char const*, int, char const*, std::__va_list)+370)
#13  pc 0x00000000005ad5d9  !libppsspp_jni.so (GenericLog(LogTypes::LOG_LEVELS, LogTypes::LOG_TYPE, char const*, int, char const*, ...)+68)
#14  pc 0x0000000000ade36d  !libppsspp_jni.so (ScreenManager::render()+64)
#15  pc 0x00000000005c1373  !libppsspp_jni.so (NativeRender(GraphicsContext*)+558)
#16  pc 0x00000000005ba157  !libppsspp_jni.so (UpdateRunLoopAndroid(_JNIEnv*)+26)
#17  pc 0x00000000005bbcd1  !libppsspp_jni.so
#18  pc 0x000000000032b88d  !libppsspp_jni.so (void* std::__ndk1::__thread_proxy<std::__ndk1::tuple<std::__ndk1::unique_ptr<std::__ndk1::__thread_struct, std::__ndk1::default_delete<std::__ndk1::__thread_struct> >, void (*)()> >(void*)+24)
#19  pc 0x00000000000aad13  /apex/com.android.runtime/lib/bionic/libc.so (__pthread_start(void*)+40)
#20  pc 0x0000000000064193  /apex/com.android.runtime/lib/bionic/libc.so (__start_thread+30)


#00  pc 0x000000000005e3e8  /apex/com.android.runtime/lib/bionic/libc.so (syscall+28)
#01  pc 0x0000000000063559  /apex/com.android.runtime/lib/bionic/libc.so (__futex_wait_ex(void volatile*, bool, int, bool, timespec const*)+92)
#02  pc 0x00000000000aa393  /apex/com.android.runtime/lib/bionic/libc.so (pthread_cond_wait+32)
#03  pc 0x0000000000bfbe49  !libppsspp_jni.so (std::__ndk1::condition_variable::wait(std::__ndk1::unique_lock<std::__ndk1::mutex>&)+12)
#04  pc 0x000000000057def7  !libppsspp_jni.so (GLRenderManager::ThreadFrame()+142)
#05  pc 0x00000000005ba235  !libppsspp_jni.so (Java_org_ppsspp_ppsspp_NativeRenderer_displayRender+64)
#06  pc 0x0000000000009053  /oat/arm/base.odex (Java_org_ppsspp_ppsspp_NativeRenderer_displayRender__+74)
at org.ppsspp.ppsspp.NativeRenderer.displayRender (Native method)
at org.ppsspp.ppsspp.NativeRenderer.onDrawFrame (NativeRenderer.java:33)
at android.opengl.GLSurfaceView$GLThread.guardedRun (GLSurfaceView.java:1591)
at android.opengl.GLSurfaceView$GLThread.run (GLSurfaceView.java:1286)

at java.lang.Object.wait (Native method)
  at java.lang.Object.wait (Object.java:442)
  at java.lang.Object.wait (Object.java:568)
  at android.opengl.GLSurfaceView$GLThread.guardedRun (GLSurfaceView.java:1515)
  at android.opengl.GLSurfaceView$GLThread.run (GLSurfaceView.java:1286)

@hrydgard
Copy link
Owner Author

hrydgard commented Jan 1, 2023

Also unrelated, another TransformUnit one, but surely also OOM:

  #00  pc 0x000000000052c05c  !libppsspp_jni.so (TransformUnit::SetDirty(SoftDirty))
  #01  pc 0x00000000005272ed  !libppsspp_jni.so (SoftGPU::Execute_Prim(unsigned int, unsigned int)+296)
  #02  pc 0x0000000000529549  !libppsspp_jni.so (SoftGPU::FastRunLoop(DisplayList&)+72)

@unknownbrackets unknownbrackets mentioned this issue Jan 1, 2023
Merged
@unknownbrackets
Copy link
Collaborator

Another, I think, GL one:

That's weird, ScreenManager::render() only logs in bad situations - transparent over nothing or no screen. That probably would just hang...

-[Unknown]

@hrydgard
Copy link
Owner Author

hrydgard commented Jan 1, 2023

That is indeed a reported hang (ANR - application not responding), not a crash. No idea how it got in that situation..

@hrydgard
Copy link
Owner Author

hrydgard commented Jan 1, 2023

Here's an oldie but goodie that have never fully gone away:

  #00  pc 0x00000000006951b4  !libppsspp_jni.so (GPUCommon::DoExecuteCall(unsigned int)+52)
  #01  pc 0x0000000000693b4c  !libppsspp_jni.so (GPUCommon::FastRunLoop(DisplayList&)+92)
  #02  pc 0x000000000069367c  !libppsspp_jni.so (GPUCommon::InterpretList(DisplayList&)+596)
  #03  pc 0x0000000000692bd0  !libppsspp_jni.so (GPUCommon::ProcessDLQueue()+100)
  #04  pc 0x0000000000693068  !libppsspp_jni.so (GPUCommon::UpdateStall(int, unsigned int)+88)
  #05  pc 0x00000000004ce294  !libppsspp_jni.so
  #06  pc 0x00000000004a7058  !libppsspp_jni.so (CallSyscallWithoutFlags(HLEFunction const*)+52)
  #07  pc 0x0000000000010eac

I'm considering throwing in an assert_msg so we can figure out which game it is happening in.. But surely it's from executing a bogus display list somehow.

@hrydgard hrydgard mentioned this issue Jan 1, 2023
Merged
@hrydgard hrydgard modified the milestones: v1.14.3, v1.15.0 Jan 2, 2023
@hrydgard
Copy link
Owner Author

hrydgard commented Jan 2, 2023

argh, looks like I caused a new kind of java exception:

Exception java.lang.RuntimeException: Can't toast on a thread that has not called Looper.prepare()
  at android.widget.Toast$TN.<init> (Toast.java:411)
  at android.widget.Toast.<init> (Toast.java:121)
  at android.widget.Toast.makeText (Toast.java:290)
  at android.widget.Toast.makeText (Toast.java:280)
  at org.ppsspp.ppsspp.NativeRenderer.onSurfaceCreated (NativeRenderer.java:60)
  at android.opengl.GLSurfaceView$GLThread.guardedRun (GLSurfaceView.java:1541)
  at android.opengl.GLSurfaceView$GLThread.run (GLSurfaceView.java:1272)

trying to display an error message:

ppsspp/android/src/org/ppsspp/ppsspp/NativeRenderer.java

Line 60 in bd2a0c9

Toast.makeText(mActivity, "Failed to initialize rendering. Close the app and try again.", Toast.LENGTH_LONG);

Oh well, at that point we're already screwed..

@hrydgard
Copy link
Owner Author

hrydgard commented Jan 2, 2023 •
edited

Also hit this one:

(GLRenderManager.h:SetBlendAndMask:801) Critical: [curRenderStep_ && curRenderStep_->stepType == GLRStepType::RENDER] *** Assertion ***

Seems the thing I added to include the game ID isn't working? darn. don't see how that's possible though? Maybe we hit a length limit?

@hrydgard
Copy link
Owner Author

hrydgard commented Jan 3, 2023 •
edited

Got a few variants of these:
art/runtime/thread.cc:1238] Native thread exited without calling DetachCurrentThread: Thread[14,tid=30521,Native,Thread*=0x9ba87900,peer=0x12dc60a0,"Thread-1467"]

I don't think Android normally asserts in this situation, might be a weird OS build. But, we are not handling thread detachment properly in all cases - getEnv() just attaches, and there's no mechanism to detach.

This one we can't do much about, though we can handle it more gracefully:

(MemArenaAndroid.cpp:Find4GBBase:160) Critical: [base != MAP_FAILED] Failed to map 256 MB of memory space: Out of memory (ULES00502 Grand Theft Auto: Vice City Stories)

A bit weird (32-bit):

  #00  pc 0x0000000000691a74  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (XXH3_hashLong_64b_default)
  #01  pc 0x00000000006916c9  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (XXH3_64bits+596)
  #02  pc 0x00000000003107db  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (JitBlockCache::FinalizeBlock(int, bool)+98)
  #03  pc 0x0000000000304f49  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (MIPSComp::ArmJit::Compile(unsigned int)+140)

@ghost
Copy link

ghost commented Jan 3, 2023

This can be considered as an issue?
Screenshot_20230103_174655

I downloaded apk from ppsspp.org and I try to update it on playstore.

@hrydgard
Copy link
Owner Author

hrydgard commented Jan 3, 2023

That's expected, you need to uninstall to switch between APK and playstore due to mismatching signing keys.

I currently don't know of a way to fix that.

@hrydgard hrydgard mentioned this issue Jan 3, 2023
Closed
@hrydgard
Copy link
Owner Author

hrydgard commented Jan 3, 2023

Here's one I haven't noticed before (as we stomp these out, I discover rarer crashes in the pile):

Thread
FORTIFY: pthread_mutex_destroy called on a destroyed mutex (0xd23f5370)
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 23399 >>> org.ppsspp.ppsspp <<<

backtrace:
  #00  pc 0x0000000000062d00  /apex/com.android.runtime/lib/bionic/libc.so (abort+172)
  #01  pc 0x00000000000ac23f  /apex/com.android.runtime/lib/bionic/libc.so (__fortify_fatal(char const*, ...)+26)
  #02  pc 0x00000000000ab979  /apex/com.android.runtime/lib/bionic/libc.so (HandleUsingDestroyedMutex(pthread_mutex_t*, char const*)+20)
  #03  pc 0x00000000000ac11f  /apex/com.android.runtime/lib/bionic/libc.so (pthread_mutex_destroy+126)
  #04  pc 0x0000000000c18971  libppsspp_jni.so (std::__ndk1::mutex::~mutex()+8)
  #05  pc 0x0000000000446433  libppsspp_jni.so (LocalFileLoader::~LocalFileLoader()+30)
  #06  pc 0x0000000000446469  libppsspp_jni.so (LocalFileLoader::~LocalFileLoader()+4)
  #07  pc 0x00000000005ce427  libppsspp_jni.so (GameInfoWorkItem::Run()+402)
  #08  pc 0x00000000005aaf11  libppsspp_jni.so
  #09  pc 0x00000000005ac0d9  libppsspp_jni.so (void* std::__ndk1::__thread_proxy<std::__ndk1::tuple<std::__ndk1::unique_ptr<std::__ndk1::__thread_struct, std::__ndk1::default_delete<std::__ndk1::__thread_struct> >, void (*)(GlobalThreadContext*, ThreadContext*), GlobalThreadContext*, ThreadContext*> >(void*)+28)
  #10  pc 0x00000000000aaf93  /apex/com.android.runtime/lib/bionic/libc.so (__pthread_start(void*)+40)
  #11  pc 0x0000000000064203  /apex/com.android.runtime/lib/bionic/libc.so (__start_thread+30)

@hrydgard
Copy link
Owner Author

hrydgard commented Jan 4, 2023

This should just be an OOM, right?

Thread
/buildbot/src/android/ndk-release-r21/external/libcxx/../../external/libcxxabi/src/abort_message.cpp:72: abort_message: assertion "terminating with uncaught exception of type std::bad_alloc: std::bad_alloc" failed
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 13497 >>> org.ppsspp.ppsspp <<<

backtrace:
  #00  pc 0x000000000001ccb6  /system/lib/libc.so (abort+58)
  #01  pc 0x000000000001cf2f  /system/lib/libc.so (__assert2+22)
  #02  pc 0x0000000000c1fe3d  armeabi_v7a.apk
  #03  pc 0x0000000000c1ff3d  armeabi_v7a.apk
  #04  pc 0x0000000000c1e4fd  armeabi_v7a.apk
  #05  pc 0x0000000000c1ddab  armeabi_v7a.apk
  #06  pc 0x0000000000c1dd73  armeabi_v7a.apk (__cxa_throw+74)
  #07  pc 0x0000000000c189dd  armeabi_v7a.apk (operator new(unsigned int)+56)
  #08  pc 0x000000000034482f  armeabi_v7a.apk (std::__ndk1::vector<unsigned char, std::__ndk1::allocator<unsigned char>>::__append(unsigned int)+100)
  #09  pc 0x00000000004665a3  armeabi_v7a.apk (SaveState::SaveToRam(std::__ndk1::vector<unsigned char, std::__ndk1::allocator<unsigned char>>&)+118)
  #10  pc 0x000000000046ba11  armeabi_v7a.apk (SaveState::StateRingbuffer::Save()+132)
  #11  pc 0x000000000046af03  armeabi_v7a.apk (SaveState::Process()+3314)
  #12  pc 0x000000000046ee77  armeabi_v7a.apk (PSP_RunLoopWhileState()+94)
  #13  pc 0x00000000005ca65d  armeabi_v7a.apk (EmuScreen::render()+220)
  #14  pc 0x0000000000adf29b  armeabi_v7a.apk (ScreenManager::render()+126)
  #15  pc 0x00000000005c20b7  armeabi_v7a.apk (NativeRender(GraphicsContext*)+558)
  #16  pc 0x00000000005bc2c7  armeabi_v7a.apk (Java_org_ppsspp_ppsspp_NativeActivity_runVulkanRenderLoop+334)
  #17  pc 0x0000000000009105  /data/app/org.ppsspp.ppsspp-ARrsgkSFQAgRLhxXT4yFDQ==/oat/arm/base.odex (org.ppsspp.ppsspp.NativeActivity.runVulkanRenderLoop+92)

@hrydgard
Copy link
Owner Author

hrydgard commented Jan 4, 2023 

backtrace:
  #00  pc 0x00000000003105c2  /data/app/org.ppsspp.ppsspp-ULG9IAVcsI4f8P_kWyxoxw==/split_config.armeabi_v7a.apk (JitBlockCache::ProxyBlock(unsigned int, unsigned int, unsigned int, unsigned char const*)+278)
  #01  pc 0x000000000030599f  /data/app/org.ppsspp.ppsspp-ULG9IAVcsI4f8P_kWyxoxw==/split_config.armeabi_v7a.apk (MIPSComp::ArmJit::ReplaceJalTo(unsigned int)+510)
  #02  pc 0x00000000002f8d79  /data/app/org.ppsspp.ppsspp-ULG9IAVcsI4f8P_kWyxoxw==/split_config.armeabi_v7a.apk (MIPSComp::ArmJit::Comp_Jump(Memory::Opcode)+204)
  #03  pc 0x0000000000456dcb  /data/app/org.ppsspp.ppsspp-ULG9IAVcsI4f8P_kWyxoxw==/split_config.armeabi_v7a.apk (MIPSCompileOp(Memory::Opcode, MIPSComp::MIPSFrontendInterface*)+166)
  #04  pc 0x0000000000305229  /data/app/org.ppsspp.ppsspp-ULG9IAVcsI4f8P_kWyxoxw==/split_config.armeabi_v7a.apk (MIPSComp::ArmJit::DoJit(unsigned int, JitBlock*)+424)
  #05  pc 0x0000000000304f3d  /data/app/org.ppsspp.ppsspp-ULG9IAVcsI4f8P_kWyxoxw==/split_config.armeabi_v7a.apk (MIPSComp::ArmJit::Compile(unsigned int)+128)
  #06  pc 0x0000000000000106

This one is actually interesting. I think we might be missing an IsFull and wipe check before we link/proxy blocks...

@hrydgard
Copy link
Owner Author

hrydgard commented Jan 4, 2023 

Thread
/buildbot/src/android/ndk-release-r21/external/libcxx/../../external/libcxxabi/src/abort_message.cpp:72: abort_message: assertion "terminating" failed
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 26034 >>> org.ppsspp.ppsspp <<<

backtrace:
  #00  pc 0x0000000000065668  /apex/com.android.runtime/lib/bionic/libc.so (abort+172)
  #01  pc 0x00000000000658d7  /apex/com.android.runtime/lib/bionic/libc.so (__assert2+22)
  #02  pc 0x0000000000c1fe3d  libppsspp_jni.so
  #03  pc 0x0000000000c1fec3  libppsspp_jni.so
  #04  pc 0x0000000000c1e4fd  libppsspp_jni.so
  #05  pc 0x0000000000c1e495  libppsspp_jni.so (std::terminate()+28)
  #06  pc 0x0000000000c1cc1d  libppsspp_jni.so (std::__ndk1::thread::~thread()+12)
  #07  pc 0x000000000047b459  libppsspp_jni.so (std::__ndk1::__shared_ptr_pointer<std::__ndk1::thread*, std::__ndk1::default_delete<std::__ndk1::thread>, std::__ndk1::allocator<std::__ndk1::thread> >::__on_zero_shared()+12)
  #08  pc 0x0000000000477f87  libppsspp_jni.so (GameManager::InstallGameOnThread(Path const&, Path const&, bool)+154)
  #09  pc 0x0000000000477e65  libppsspp_jni.so (GameManager::Update()+624)
  #10  pc 0x00000000005c1ead  libppsspp_jni.so (NativeRender(GraphicsContext*)+36)
  #11  pc 0x00000000005bae57  libppsspp_jni.so (UpdateRunLoopAndroid(_JNIEnv*)+26)
  #12  pc 0x00000000005bc9e1  libppsspp_jni.so
  #13  pc 0x000000000032bbcd  libppsspp_jni.so (void* std::__ndk1::__thread_proxy<std::__ndk1::tuple<std::__ndk1::unique_ptr<std::__ndk1::__thread_struct, std::__ndk1::default_delete<std::__ndk1::__thread_struct> >, void (*)()> >(void*)+24)
  #14  pc 0x00000000000b0567  /apex/com.android.runtime/lib/bionic/libc.so (__pthread_start(void*)+40)
  #15  pc 0x0000000000066b37  /apex/com.android.runtime/lib/bionic/libc.so (__start_thread+30)

@unknownbrackets unknownbrackets mentioned this issue Jan 5, 2023
Merged
@hrydgard
Copy link
Owner Author

hrydgard commented Jan 9, 2023

here's one curious assert that we probably should try to avoid crashing on (omitting the callstack because uninteresting):

(VulkanFrameData.cpp:AcquireNextImage:80): [false] (ULES00502 Grand Theft Auto: Vice City Stories) vkAcquireNextImageKHR failed! result=VK_TIMEOUT

hrydgard added a commit that referenced this issue Jan 12, 2023
@hrydgard
Handle a couple of valid but rare return values from vkAcquireNextIma…
609e7f1 
…geKHR.

See #16601
@unknownbrackets
Copy link
Collaborator

Are all the most common crashes captured here / potentially resolved? Should we close this for next time around?

-[Unknown]

@hrydgard
Copy link
Owner Author

hrydgard commented Feb 12, 2023 •
edited

Except for the OOMs which are hard to do much about, enough of it is taken care of for this time around, yeah.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v1.15.0
Development

No branches or pull requests
2 participants
@hrydgard @unknownbrackets