Runtime tests have issues finding the testing app on Android x64

[VincentBu]

Run: runtime 20210526.87

Failed tests:

Mono Android x64 Release @ Ubuntu.1804.Amd64.Android.Open
- JIT/Methodical/*

Also failed in :

Mono Android x64 Release @ Ubuntu.1804.Amd64.Android.Open
- JIT/Regression/JitBlue/DevDiv_461649/DevDiv_461649/DevDiv_461649.sh
- Interop/StructMarshalling/ReversePInvoke/MarshalSeqStruct/DelegatePInvoke/DelegatePInvokeTest/DelegatePInvokeTest.sh
- GC/Scenarios/Dynamo/dynamo/dynamo.sh

One of Error messages:

Return code:      78
Raw output file:      /datadisks/disk1/work/A0E608D3/w/B0710944/e/JIT/Methodical/Reports/JIT.Methodical/VT/callconv/_il_reljumper4/_il_reljumper4.output.txt
Raw output:
BEGIN EXECUTION
XHarness command issued: android test --instrumentation=net.dot.MonoRunner --package-name=net.dot.JIT_Methodical --app=/datadisks/disk1/work/A0E608D3/w/B0710944/e/JIT/Methodical/JIT_Methodical.apk --output-directory=/datadisks/disk1/work/A0E608D3/w/B0710944/e/JIT/Methodical/VT/callconv/_il_reljumper4 --arg=entrypoint:libname=_il_reljumper4.dll --expected-exit-code=100 -v
[23:00:51] dbug: Android Test command called: App = /datadisks/disk1/work/A0E608D3/w/B0710944/e/JIT/Methodical/JIT_Methodical.apk
Instrumentation Name = net.dot.MonoRunner
[23:00:51] dbug: Output Directory:/datadisks/disk1/work/A0E608D3/w/B0710944/e/JIT/Methodical/VT/callconv/_il_reljumper4
Timeout = 900 seconds.
[23:00:51] dbug: Arguments to instrumentation:
[23:00:51] dbug: ADBRunner using ADB.exe supplied from /datadisks/disk1/work/A0E608D3/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21271.1/tools/net6.0/any/../../../runtimes/any/native/adb/linux/adb
[23:00:51] dbug: Full resolved path:'/datadisks/disk1/work/A0E608D3/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21271.1/runtimes/any/native/adb/linux/adb'
[23:00:51] info: Will attempt to run device on detected architecture: 'x86_64'
[23:00:51] dbug: Executing command: '/datadisks/disk1/work/A0E608D3/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21271.1/runtimes/any/native/adb/linux/adb  start-server'
[23:00:51] dbug:
[23:00:51] dbug: Executing command: '/datadisks/disk1/work/A0E608D3/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21271.1/runtimes/any/native/adb/linux/adb  devices -l'
[23:00:51] dbug: Evaluating output line for device serial: emulator-5554          device product:sdk_phone_x86 model:Android_SDK_built_for_x86 device:generic_x86 transport_id:1
[23:00:51] dbug: Executing command: '/datadisks/disk1/work/A0E608D3/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21271.1/runtimes/any/native/adb/linux/adb  -s emulator-5554 shell getprop ro.product.cpu.abilist'
[23:00:51] dbug: Evaluating output line for device serial: emulator-5556          device product:sdk_phone_x86_64 model:Android_SDK_built_for_x86_64 device:generic_x86_64 transport_id:2
[23:00:51] dbug: Executing command: '/datadisks/disk1/work/A0E608D3/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21271.1/runtimes/any/native/adb/linux/adb  -s emulator-5556 shell getprop ro.product.cpu.abilist'
[23:00:51] dbug: Using first-found compatible device of 1 total- serial: 'emulator-5556' - architecture: x86_64,x86
[23:00:51] info: Active Android device set to serial 'emulator-5556'
[23:00:51] info: Waiting for device to be available (max 5 minutes)
[23:00:51] dbug: Executing command: '/datadisks/disk1/work/A0E608D3/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21271.1/runtimes/any/native/adb/linux/adb -s emulator-5556 wait-for-device'
[23:00:51] dbug:
[23:00:51] dbug: Executing command: '/datadisks/disk1/work/A0E608D3/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21271.1/runtimes/any/native/adb/linux/adb -s emulator-5556 shell getprop sys.boot_completed'
[23:00:51] dbug: Waited 0.0216012 seconds for device for sys.boot_completed to be 1.
[23:00:51] dbug: Executing command: '/datadisks/disk1/work/A0E608D3/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21271.1/runtimes/any/native/adb/linux/adb -s emulator-5556 version'
[23:00:51] dbug: Working with Android Debug Bridge version 1.0.41
Version 29.0.6-6198805
Installed as /datadisks/disk1/work/A0E608D3/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21271.1/runtimes/any/native/adb/linux/adb

[23:00:51] info: Attempting to remove apk 'net.dot.JIT_Methodical':
[23:00:51] dbug: Executing command: '/datadisks/disk1/work/A0E608D3/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21271.1/runtimes/a


Stack trace
   at JIT_Methodical._VT_callconv__il_reljumper4__il_reljumper4_._VT_callconv__il_reljumper4__il_reljumper4_sh()

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[VincentBu]

Failed again in runtime 20210608.73 with JIT.Regression tests

One of Error message:

Return code:      85
Raw output file:      /datadisks/disk1/work/A9920943/w/97010892/uploads/Reports/JIT.Regression/JitBlue/Runtime_41100/Runtime_41100/Runtime_41100.output.txt
Raw output:
BEGIN EXECUTION
XHarness command issued: android test --instrumentation=net.dot.MonoRunner --package-name=net.dot.JIT_Regression --app=/datadisks/disk1/work/A9920943/w/97010892/e/JIT/Regression/JIT_Regression.apk --output-directory=/datadisks/disk1/work/A9920943/w/97010892/uploads/Reports/JIT.Regression/JitBlue/Runtime_41100/Runtime_41100 --arg=entrypoint:libname=Runtime_41100.dll --expected-exit-code=100 -v
[20:07:21] dbug: Android Test command called: App = /datadisks/disk1/work/A9920943/w/97010892/e/JIT/Regression/JIT_Regression.apk
Instrumentation Name = net.dot.MonoRunner
[20:07:21] dbug: Output Directory:/datadisks/disk1/work/A9920943/w/97010892/uploads/Reports/JIT.Regression/JitBlue/Runtime_41100/Runtime_41100
Timeout = 900 seconds.
[20:07:21] dbug: Arguments to instrumentation:
[20:07:21] dbug: ADBRunner using ADB.exe supplied from /datadisks/disk1/work/A9920943/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21307.1/tools/net6.0/any/../../../runtimes/any/native/adb/linux/adb
[20:07:21] dbug: Full resolved path:'/datadisks/disk1/work/A9920943/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21307.1/runtimes/any/native/adb/linux/adb'
[20:07:21] info: Will attempt to run device on detected architecture: 'x86_64'
[20:07:21] dbug: Executing command: '/datadisks/disk1/work/A9920943/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21307.1/runtimes/any/native/adb/linux/adb  start-server'
[20:07:21] dbug:
[20:07:21] dbug: Executing command: '/datadisks/disk1/work/A9920943/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21307.1/runtimes/any/native/adb/linux/adb  devices -l'
[20:07:21] dbug: Evaluating output line for device serial: emulator-5556          device product:sdk_phone_x86 model:Android_SDK_built_for_x86 device:generic_x86 transport_id:2
[20:07:21] dbug: Executing command: '/datadisks/disk1/work/A9920943/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21307.1/runtimes/any/native/adb/linux/adb  -s emulator-5556 shell getprop ro.product.cpu.abilist'
[20:07:21] fail: No devices with architecture 'x86_64' was found among attached and authorized devices.
[20:07:21] crit: Failed to find compatible device: x86_64
Microsoft.DotNet.XHarness.Common.CLI.NoDeviceFoundException: Failed to find compatible device: x86_64
at Microsoft.DotNet.XHarness.CLI.Commands.Android.AndroidInstallCommand.InvokeHelper(ILogger logger, String apkPackageName, String appPackagePath, IEnumerable`1 apkRequiredArchitecture, String deviceId, TimeSpan bootTimeoutSeconds, AdbRunner runner) in //src/Microsoft.DotNet.XHarness.CLI/Commands/Android/AndroidInstallCommand.cs:line 104
at Microsoft.DotNet.XHarness.CLI.Commands.Android.AndroidTestCommand.InvokeInternal(ILogger logger) in //src/Microsoft.DotNet.XHarness.CLI/Commands/Android/AndroidTestCommand.cs:line 78
XHarness exit code: 85 (ADB_DEVICE_ENUMERATION_FAILURE)
Test Harness Exitcode is : 85
To run the test:

set CORE_ROOT=/datadisks/disk1/work/A9920943/p
/datadisks/disk1/work/A9920943/w/97010892/e/JIT/Regression/JitBlue/Runtime_41100/Runtime_41100/Runtime_41100.sh
Expected: True
Actual:   False


Stack trace
   at JIT_Regression._JitBlue_Runtime_41100_Runtime_41100_Runtime_41100_._JitBlue_Runtime_41100_Runtime_41100_Runtime_41100_sh()

[SamMonoRT]

@fanyang-mono : another intermittent failure on Android "Mono Android x64 Release @ Ubuntu.1804.Amd64.Android.Open" lane

[fanyang-mono]

Both of the log looks like issues related to the xharness test API. The first one stuck at uninstalling Android app. The second one showed that it couldn't find the Android emulator. @premun @greenEkatherine Could your team take a look at these failures?

[greenEkatherine]

I'm on it @fanyang-mono

[VincentBu]

Failed again in runtime 20210629.108 with JIT.jit64.* reflection.* Loader/regressions/GitHub_15544/main/main.sh CoreMangLib/system/environment/environment_version/environment_version.sh tests.

Failed tests:

Mono Android x64 Release @ Ubuntu.1804.Amd64.Android.Open

- JIT/jit64/*
- reflection/regression/manyStackArgs/manyStackArgs/manyStackArgs.sh
- reflection/DefaultInterfaceMethods/GetInterfaceMapConsumer/GetInterfaceMapConsumer.sh
- Loader/regressions/GitHub_15544/main/main.sh
- CoreMangLib/system/environment/environment_version/environment_version.sh

One of Error messages:

Return code:      85
Raw output file:      /datadisks/disk1/work/A9CD0924/w/AF4909B5/uploads/Reports/JIT.jit64/opt/cg/cgstress/CgStress1_do/CgStress1_do.output.txt
Raw output:
BEGIN EXECUTION
XHarness command issued: android run --instrumentation=net.dot.MonoRunner --package-name=net.dot.JIT_jit64 --output-directory=/datadisks/disk1/work/A9CD0924/w/AF4909B5/uploads/Reports/JIT.jit64/opt/cg/cgstress/CgStress1_do --arg=entrypoint:libname=CgStress1_do.dll --expected-exit-code=100 -v
[23:48:11] dbug: Android Run command called: App = net.dot.JIT_jit64

[23:48:11] dbug: Timeout = 900 seconds.
[23:48:11] dbug: ADBRunner using ADB.exe supplied from /datadisks/disk1/work/A9CD0924/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21324.2/tools/net6.0/any/../../../runtimes/any/native/adb/linux/adb
[23:48:11] dbug: Full resolved path:'/datadisks/disk1/work/A9CD0924/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21324.2/runtimes/any/native/adb/linux/adb'
[23:48:11] dbug: Executing command: '/datadisks/disk1/work/A9CD0924/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21324.2/runtimes/any/native/adb/linux/adb  start-server'
[23:48:11] dbug:
[23:48:11] dbug: Executing command: '/datadisks/disk1/work/A9CD0924/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21324.2/runtimes/any/native/adb/linux/adb  devices -l'
[23:48:11] dbug: Evaluating output line for device serial: emulator-5554          device product:sdk_phone_x86_64 model:Android_SDK_built_for_x86_64 device:generic_x86_64 transport_id:3
[23:48:11] dbug: Executing command: '/datadisks/disk1/work/A9CD0924/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21324.2/runtimes/any/native/adb/linux/adb  -s emulator-5554 shell pm list packages -3'
[23:48:11] dbug: Evaluating output line for device serial: emulator-5556          device product:sdk_phone_x86 model:Android_SDK_built_for_x86 device:generic_x86 transport_id:2
[23:48:11] dbug: Executing command: '/datadisks/disk1/work/A9CD0924/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21324.2/runtimes/any/native/adb/linux/adb  -s emulator-5556 shell pm list packages -3'
[23:48:11] fail: No devices with app 'package:net.dot.JIT_jit64' was found among attached and authorized devices.
[23:48:11] fail: Cannot find a device with app=package:net.dot.JIT_jit64, please check that a device is attached
XHarness exit code: 85 (ADB_DEVICE_ENUMERATION_FAILURE)
Test Harness Exitcode is : 85
To run the test:

set CORE_ROOT=/datadisks/disk1/work/A9CD0924/p
/datadisks/disk1/work/A9CD0924/w/AF4909B5/e/JIT/jit64/opt/cg/cgstress/CgStress1_do/CgStress1_do.sh
Expected: True
Actual:   False


Stack trace
   at JIT_jit64._opt_cg_cgstress_CgStress1_do_CgStress1_do_._opt_cg_cgstress_CgStress1_do_CgStress1_do_sh() in JIT.jit64.XUnitWrapper.dll:token 0x6000007+0x2b2

[radical]

I'm not sure if this is the same failure. JIT.Methodical.a-dA-D seems to timeout on CoreCLR Pri0 Runtime Tests Run windows arm64 checked:

build: https://dev.azure.com/dnceng/public/_build/results?buildId=1213972&view=logs&jobId=bb94f37e-a09b-5b51-09b2-ab78e7aa5103&j=bb94f37e-a09b-5b51-09b2-ab78e7aa5103&t=506cadc9-3cc4-5d96-6847-011f75a33382
PR: dotnet/runtime#54936

The console log has only this line:
Console log: 'JIT.Methodical.a-dA-D' from job dc9b8409-2ae7-410d-81eb-24a7d8b741c2 workitem ae895520-db5a-4e94-8fdd-b661552c6c93 (windows.10.arm64v8.open) executed on machine DDARM64-119

[fanyang-mono]

@radical This issue tracks Android specific issues. The one you are seeing on windows should be tracked by a different issue.

[VincentBu]

Failed again in runtime 20210706.106
Failed tests

Mono Android x64 Release @ Ubuntu.1804.Amd64.Android.Open

- JIT/Methodical/*
- JIT/Math/Functions/Functions_r/Functions_r.sh
- JIT/Math/Functions/Functions_ro/Functions_ro.sh
- Regressions/coreclr/*
- baseservices/*

One of error messages

Return code:      85
Raw output file:      /datadisks/disk1/work/AC000986/w/B8490A0C/uploads/Reports/JIT.Methodical/int64/signed/_rels_ldsfld_mulovf/_rels_ldsfld_mulovf.output.txt
Raw output:
BEGIN EXECUTION
XHarness command issued: android run --instrumentation=net.dot.MonoRunner --package-name=net.dot.JIT_Methodical --output-directory=/datadisks/disk1/work/AC000986/w/B8490A0C/uploads/Reports/JIT.Methodical/int64/signed/_rels_ldsfld_mulovf --arg=entrypoint:libname=_rels_ldsfld_mulovf.dll --expected-exit-code=100 -v
[03:27:56] dbug: Android Run command called: App = net.dot.JIT_Methodical

[03:27:56] dbug: Timeout = 900 seconds.
[03:27:56] dbug: ADBRunner using ADB.exe supplied from /datadisks/disk1/work/AC000986/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21330.2/tools/net6.0/any/../../../runtimes/any/native/adb/linux/adb
[03:27:56] dbug: Full resolved path:'/datadisks/disk1/work/AC000986/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21330.2/runtimes/any/native/adb/linux/adb'
[03:27:56] dbug: Executing command: '/datadisks/disk1/work/AC000986/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21330.2/runtimes/any/native/adb/linux/adb  start-server'
[03:27:57] dbug:
[03:27:57] dbug: Executing command: '/datadisks/disk1/work/AC000986/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21330.2/runtimes/any/native/adb/linux/adb  devices -l'
[03:27:57] dbug: Evaluating output line for device serial: emulator-5554          device product:sdk_phone_x86_64 model:Android_SDK_built_for_x86_64 device:generic_x86_64 transport_id:9
[03:27:57] dbug: Executing command: '/datadisks/disk1/work/AC000986/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21330.2/runtimes/any/native/adb/linux/adb  -s emulator-5554 shell pm list packages -3'
[03:27:57] dbug: Evaluating output line for device serial: emulator-5556          device product:sdk_phone_x86 model:Android_SDK_built_for_x86 device:generic_x86 transport_id:2
[03:27:57] dbug: Executing command: '/datadisks/disk1/work/AC000986/p/microsoft.dotnet.xharness.cli/1.0.0-prerelease.21330.2/runtimes/any/native/adb/linux/adb  -s emulator-5556 shell pm list packages -3'
[03:27:57] fail: No devices with app 'package:net.dot.JIT_Methodical' was found among attached and authorized devices.
[03:27:57] fail: Cannot find a device with app=package:net.dot.JIT_Methodical, please check that a device is attached
XHarness exit code: 85 (ADB_DEVICE_ENUMERATION_FAILURE)
Test Harness Exitcode is : 85
To run the test:

set CORE_ROOT=/datadisks/disk1/work/AC000986/p
/datadisks/disk1/work/AC000986/w/B8490A0C/e/JIT/Methodical/int64/signed/_rels_ldsfld_mulovf/_rels_ldsfld_mulovf.sh
Expected: True
Actual:   False


Stack trace
   at JIT_Methodical._int64_signed__rels_ldsfld_mulovf__rels_ldsfld_mulovf_._int64_signed__rels_ldsfld_mulovf__rels_ldsfld_mulovf_sh() in JIT.Methodical.XUnitWrapper.dll:token 0x60003e5+0x2b2

[SamMonoRT]

@greenEkatherine @premun - should this be moved to the dotnet\xharness repo for tracking ?

[greenEkatherine]

@SamMonoRT nice idea, I'm going to transfer it there

[premun]

Quick glance it seems like the installation of the app failed because there was no space on the device so probably the runtime tests should fail there and not try further? @fanyang-mono?

JIT_jit64_install.output.txt

[23:48:10] fail: Error:
                 Exit code: 1
                 Standard Output:
                 Performing Streamed Install
                 
                 Standard Error:
                 adb: failed to install /datadisks/disk1/work/A9CD0924/w/AF4909B5/e/JIT/jit64/JIT_jit64.apk: 
                 Exception occurred while executing:
                 android.os.ParcelableException: java.io.IOException: Requested internal only, but not enough space
                 	at android.util.ExceptionUtils.wrap(ExceptionUtils.java:34)
                 	at com.android.server.pm.PackageInstallerService.createSession(PackageInstallerService.java:467)
                 	at com.android.server.pm.PackageManagerShellCommand.doCreateSession(PackageManagerShellCommand.java:2700)
                 	at com.android.server.pm.PackageManagerShellCommand.runInstall(PackageManagerShellCommand.java:1061)
                 	at com.android.server.pm.PackageManagerShellCommand.onCommand(PackageManagerShellCommand.java:169)
                 	at android.os.ShellCommand.exec(ShellCommand.java:104)
                 	at com.android.server.pm.PackageManagerService.onShellCommand(PackageManagerService.java:21742)
                 	at android.os.Binder.shellCommand(Binder.java:881)
                 	at android.os.Binder.onTransact(Binder.java:765)
                 	at android.content.pm.IPackageManager$Stub.onTransact(IPackageManager.java:4860)
                 	at com.android.server.pm.PackageManagerService.onTransact(PackageManagerService.java:4016)
                 	at android.os.Binder.execTransactInternal(Binder.java:1021)
                 	at android.os.Binder.execTransact(Binder.java:994)
                 Caused by: java.io.IOException: Requested internal only, but not enough space
                 	at com.android.internal.content.PackageHelper.resolveInstallVolume(PackageHelper.java:234)
                 	at com.android.internal.content.PackageHelper.resolveInstallVolume(PackageHelper.java:150)
                 	at com.android.internal.content.PackageHelper.resolveInstallVolume(PackageHelper.java:165)
                 	at com.android.server.pm.PackageInstallerService.createSessionInternal(PackageInstallerService.java:571)
                 	at com.android.server.pm.PackageInstallerService.createSession(PackageInstallerService.java:465)
                 	... 11 more
                 
                 
[23:48:10] crit: Install failure: Test command cannot continue
[23:48:10] info: Attempting to remove apk 'net.dot.JIT_jit64': 
[23:48:10] info: APK 'net.dot.JIT_jit64' not on device.
XHarness exit code: 78 (PACKAGE_INSTALLATION_FAILURE)

@greenEkatherine we're tracking this somewhere already, right?

[greenEkatherine]

@premun yes, for issues with 78 exit code and message with "not enough space" either internal or external we raised the issue #660
Fix was rolled out yesterday so it should mitigate on new machines.

But also I see there are few logs with 85 exit codes which means we cannot find a device. I need to check it separately

[premun]

@greenEkatherine my point was that it seems like the runtime tests call the install command but ignore the exit code and then they proceed to call the run commands which then return "device not found" since there's no device with the app?

So I think the exit code can be misleading?

And we should probably stop the execution if we fail the installation?

[greenEkatherine]

ah, got it, nice catch! @premun you are right, tests failed with exit code == 85 because installation on previous step failed and we cannot find a device with app installed

It means that fix for #660 should solve the issue from our side. @fanyang-mono could you please add a check for an exit code from install command and stop if it fails?

[fanyang-mono]

Yes, I will add that check.

[greenEkatherine]

Check was added and related issue was resolved, closing