Instrumentation Runner crashing in x86 UI tests: JNI error crash in GeckoView

[AaronMT]

https://console.firebase.google.com/u/0/project/moz-fenix/testlab/histories/bh.66b7091e15d53d45/matrices/8095517053815793461

06-13 14:27:43.208: E/lla.fenix.debu(10168): JNI ERROR (app bug): accessed stale Local 0xe5e5e5e5  (index 241065566 in a table of size 6)

Here's the first 20 frames

06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542] JNI DETECTED ERROR IN APPLICATION: use of deleted local reference 0xe5e5e5e5
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]     from long org.mozilla.gecko.GeckoThread.runUiThreadCallback()
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542] "main" prio=5 tid=1 Runnable
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]   | group="main" sCount=0 dsCount=0 flags=0 obj=0x74571ee0 self=0xe54e2000
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]   | sysTid=10168 nice=-10 cgrp=default sched=0/0 handle=0xea11d494
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]   | state=R schedstat=( 7459577644 3443665154 5045 ) utm=724 stm=21 core=0 HZ=100
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]   | stack=0xff4a2000-0xff4a4000 stackSize=8MB
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]   | held mutexes= "mutator lock"(shared held)
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]   native: #00 pc 004152f6  /system/lib/libart.so (art::DumpNativeStack(std::__1::basic_ostream<char, std::__1::char_traits<char>>&, int, BacktraceMap*, char const*, art::ArtMethod*, void*, bool)+198)
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]   native: #01 pc 0051048e  /system/lib/libart.so (art::Thread::DumpStack(std::__1::basic_ostream<char, std::__1::char_traits<char>>&, bool, BacktraceMap*, bool) const+382)
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]   native: #02 pc 0050b743  /system/lib/libart.so (art::Thread::Dump(std::__1::basic_ostream<char, std::__1::char_traits<char>>&, bool, BacktraceMap*, bool) const+83)
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]   native: #03 pc 0031a8b0  /system/lib/libart.so (art::JavaVMExt::JniAbort(char const*, char const*)+1088)
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]   native: #04 pc 0031adc5  /system/lib/libart.so (art::JavaVMExt::JniAbortF(char const*, char const*, ...)+117)
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]   native: #05 pc 00516c77  /system/lib/libart.so (art::Thread::DecodeJObject(_jobject*) const+871)
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]   native: #06 pc 000d5559  /system/lib/libart.so (art::(anonymous namespace)::ScopedCheck::CheckInstance(art::ScopedObjectAccess&, art::(anonymous namespace)::ScopedCheck::InstanceKind, _jobject*, bool)+153)
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]   native: #07 pc 000d47b7  /system/lib/libart.so (art::(anonymous namespace)::ScopedCheck::CheckPossibleHeapValue(art::ScopedObjectAccess&, char, art::(anonymous namespace)::JniValueType)+551)
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]   native: #08 pc 000d3bdb  /system/lib/libart.so (art::(anonymous namespace)::ScopedCheck::Check(art::ScopedObjectAccess&, bool, char const*, art::(anonymous namespace)::JniValueType*)+811)
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]   native: #09 pc 000d6fa7  /system/lib/libart.so (art::(anonymous namespace)::CheckJNI::NewRef(char const*, _JNIEnv*, _jobject*, art::IndirectRefKind)+903)
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]   native: #10 pc 000c030b  /system/lib/libart.so (art::(anonymous namespace)::CheckJNI::NewGlobalRef(_JNIEnv*, _jobject*)+43)
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]   native: #11 pc 0286e70d  /data/app/org.mozilla.fenix.debug-crc_zILjlgZ8dhHr3XDXSQ==/lib/x86/libxul.so (offset fb000) (txXSLKey::testNode(txXPathNode const&, txKeyValueHashKey&, nsTHashtable<txKeyValueHashEntry>&, txExecutionState&)+285)
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]   native: #12 pc 0286e645  /data/app/org.mozilla.fenix.debug-crc_zILjlgZ8dhHr3XDXSQ==/lib/x86/libxul.so (offset fb000) (txXSLKey::testNode(txXPathNode const&, txKeyValueHashKey&, nsTHashtable<txKeyValueHashEntry>&, txExecutionState&)+85)
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]   native: #13 pc 00fea712  /data/app/org.mozilla.fenix.debug-crc_zILjlgZ8dhHr3XDXSQ==/lib/x86/libxul.so (offset fb000) (mozilla::gfx::DuplicateEdges(mozilla::gfx::DataSourceSurface*, mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const&)+1170)
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]   native: #14 pc 009bfe69  /data/app/org.mozilla.fenix.debug-crc_zILjlgZ8dhHr3XDXSQ==/lib/x86/libxul.so (offset fb000) (void mozilla::ipc::WriteIPDLParam<mozilla::dom::HeadersGuardEnum const&>(IPC::Message*, mozilla::ipc::IProtocol*, mozilla::dom::HeadersGuardEnum const&&&)+25)
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]   native: #15 pc 00783a3a  /data/app/org.mozilla.fenix.debug-crc_zILjlgZ8dhHr3XDXSQ==/lib/x86/libxul.so (offset fb000) (non-virtual thunk to nsFtpState::QueryInterface(nsID const&, void**)+26)
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]   native: #16 pc 007828f0  /data/app/org.mozilla.fenix.debug-crc_zILjlgZ8dhHr3XDXSQ==/lib/x86/libxul.so (offset fb000) (nsFtpChannel::OpenContentStream(bool, nsIInputStream**, nsIChannel**)+64)
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]   native: #17 pc 00783044  /data/app/org.mozilla.fenix.debug-crc_zILjlgZ8dhHr3XDXSQ==/lib/x86/libxul.so (offset fb000) (nsFtpChannel::GetStatusArg(nsresult, nsTString<char16_t>&)+20)
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]   native: #18 pc 00783383  /data/app/org.mozilla.fenix.debug-crc_zILjlgZ8dhHr3XDXSQ==/lib/x86/libxul.so (offset fb000) (nsFtpChannel::Suspend()+163)
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]   native: #19 pc 0283e86a  /data/app/org.mozilla.fenix.debug-crc_zILjlgZ8dhHr3XDXSQ==/lib/x86/libxul.so (offset fb000) (mozilla::dom::AndroidWebAuthnResult::~AndroidWebAuthnResult()+234)
06-13 14:27:43.734: A/lla.fenix.debu(10168): java_vm_ext.cc:542]   native: #20 pc 0285dcb8  /data/app/org.mozilla.fenix.debug-crc_zILjlgZ8dhHr3XDXSQ==/lib/x86/libxul.so (offset fb000) (txNodeSet::copyElements(txXPathNode*, txXPathNode const*, txXPathNode const*)+56)

┆Issue is synchronized with this Jira Task

[AaronMT]

Couldn't tell you to whom looks at Gecko crashes, or where one might file this so please triage appropriately. E.g, is this GFX based on frame 13?

[AaronMT]

A number of tests are affected as they traverse the application using similar interfaces such as tab exploration. They are crashing with the same JNI Error:

06-17 13:15:20.404: A/lla.fenix.debu(9984): java_vm_ext.cc:542] JNI DETECTED ERROR IN APPLICATION: use of deleted local reference 0xe5e5e5e5

GeckoView Bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1644561

[sv-ohorvath]

Disabled for now the most frequent failures:
threeDotMenuItemsTest
verifyAboutFirefoxPreview
testStrictVisitDisableExceptionToggle

[sv-ohorvath]

Found one more crashing test:
HomeScreenTest.privateModeScreenItemsTest: https://console.firebase.google.com/u/0/project/moz-fenix/testlab/histories/bh.66b7091e15d53d45/matrices/5904914694260524373/executions/bs.8719d13f41560240/test-cases
PR to disable it: #11768

[kbrosnan]

This should be fixed in nightly builds. Looks like we need to backout a couple tests that were disabled.