One time CI/CD with Quick Mode on Browser Stack / Firebase to check which devices work on which benchmarks

[Mostelk]

Per @swasson488 , we Need check which devices on Firebase / Browser-Stack still work with latest benchmark to check supported device list, and which benchmarks are supported

[anhappdev]

Here is the list of device, grouped by Android version, extracted from
https://www.browserstack.com/list-of-browsers-and-platforms/automate

Android 15.0
	•	Samsung Galaxy S25 Ultra
	•	Samsung Galaxy S25
	•	Google Pixel 9 Pro XL
	•	Google Pixel 9 Pro
	•	Google Pixel 9
	•	Google Pixel 6 Pro
	•	Samsung Galaxy Tab S10 Plus

⸻

Android 14.0
	•	Samsung Galaxy S24 Ultra
	•	Samsung Galaxy S24
	•	Google Pixel 8 Pro
	•	Google Pixel 8
	•	OnePlus 12R
	•	Samsung Galaxy Tab A9 Plus

⸻

Android 13.0
	•	Samsung Galaxy S23 Ultra
	•	Samsung Galaxy S23
	•	Google Pixel 7 Pro
	•	Google Pixel 7
	•	Google Pixel 6 Pro
	•	OnePlus 11R
	•	Samsung Galaxy Tab S9

⸻

Android 12.0
	•	Samsung Galaxy S22 Ultra
	•	Samsung Galaxy S22 Plus
	•	Samsung Galaxy S22
	•	Samsung Galaxy S21
	•	Google Pixel 6 Pro
	•	Google Pixel 6
	•	Google Pixel 5
	•	Huawei Nova 11 SE
	•	Samsung Galaxy Tab S8

⸻

Android 11.0
	•	Samsung Galaxy S21 Ultra
	•	Samsung Galaxy S21
	•	Samsung Galaxy S21 Plus
	•	Samsung Galaxy M52
	•	Samsung Galaxy M32
	•	Samsung Galaxy A52
	•	Google Pixel 5
	•	OnePlus 9
	•	Xiaomi Redmi Note 11
	•	Vivo Y21
	•	Vivo V21
	•	Oppo Reno 6
	•	Oppo A96
	•	Motorola Moto G71 5G
	•	Samsung Galaxy Tab S7

⸻

Android 10.0
	•	Samsung Galaxy S20
	•	Samsung Galaxy S20 Plus
	•	Samsung Galaxy S20 Ultra
	•	Samsung Galaxy Note 20
	•	Samsung Galaxy A51
	•	Samsung Galaxy A11
	•	Google Pixel 4 XL
	•	Google Pixel 4
	•	OnePlus 8
	•	OnePlus 7T
	•	Xiaomi Redmi Note 9
	•	Vivo Y50
	•	Oppo Reno 3 Pro
	•	Motorola Moto G9 Play
	•	Samsung Galaxy Tab S7

⸻

Android 9.0
	•	Samsung Galaxy S10
	•	Samsung Galaxy A10
	•	Google Pixel 3
	•	OnePlus 7
	•	Xiaomi Redmi Note 8
	•	Motorola Moto G7 Play
	•	Huawei P30
	•	Samsung Galaxy Tab S6

⸻

Android 8.1
	•	Samsung Galaxy Note 9

⸻

Android 8.0
	•	Samsung Galaxy S9

⸻

Android 7.0
	•	Samsung Galaxy S8

⸻

[anhappdev]

Here is the result of the runs. Please note that our application supports Android 11.0 and later.

Samsung

Device Name	Android Version	Status	SOC	Model Code
Galaxy S25	15.0	✅ Success	Qualcomm SM8750-AB Snapdragon 8 Elite (3 nm)	SM-S931B
Galaxy S25 Ultra	15.0	✅ Success	Qualcomm SM8750-AB Snapdragon 8 Elite (3 nm)	SM-S938B
Galaxy Tab S10 Plus	15.0	❌ Failure	MediaTek MT6989	SM-X826B
Galaxy S24 Ultra	14.0	✅ Success	Qualcomm SM8650-AC Snapdragon 8 Gen 3 (4 nm)	SM-S928B
Galaxy S24	14.0	✅ Success	Qualcomm SM8650-AC Snapdragon 8 Gen 3 (4 nm) - USA/Canada/China OR Exynos 2400 (4 nm) - International	SM-S921B
Galaxy Tab A9 Plus	14.0	❌ Failure	Qualcomm SM6375 Snapdragon 695 5G (6 nm)	SM-X216B
Galaxy S23 Ultra	13.0	✅ Success	Qualcomm SM8550-AC Snapdragon 8 Gen 2 (4 nm)	SM-S918B
Galaxy S23	13.0	✅ Success	Qualcomm SM8550-AC Snapdragon 8 Gen 2 (4 nm)	SM-S911B
Galaxy Tab S9	13.0	✅ Success	Qualcomm SM8550-AC Snapdragon 8 Gen 2 (4 nm)	SM-X716B
Galaxy S22 Plus	12.0	✅ Success	Exynos 2200 (4 nm) - Europe OR Qualcomm SM8450 Snapdragon 8 Gen 1 (4 nm) - ROW	SM-S906B
Galaxy S22 Ultra	12.0	✅ Success	Exynos 2200 (4 nm) - Europe OR Qualcomm SM8450 Snapdragon 8 Gen 1 (4 nm) - ROW	SM-S908B
Galaxy S22	12.0	✅ Success	Exynos 2200 (4 nm) - Europe OR Qualcomm SM8450 Snapdragon 8 Gen 1 (4 nm) - ROW	SM-S901B
Galaxy Tab S8	12.0	❌ Failure	Qualcomm SM8450 Snapdragon 8 Gen 1 (4 nm)	SM-X706B
Galaxy S21	12.0, 11.0	✅ Success	Samsung Exynos 2100	SM-G991B
Galaxy S21 Ultra	11.0	✅ Success	Samsung Exynos 2100	SM-G998B
Galaxy S21 Plus	11.0	✅ Success	Samsung Exynos 2100	SM-G996B
Galaxy M52	11.0	✅ Success	Qualcomm Snapdragon 7325	SM-M526B
Galaxy A52	11.0	❌ Failure	Qualcomm SM7125 Snapdragon 720G (8 nm)	SM-A525F
Galaxy M32	11.0	✅ Success	MediaTek MT6853V/NZA	SM-M326B
Galaxy Tab S7	11.0	❌ Failure	Qualcomm SM8250 Snapdragon 865 5G+ (7 nm+)	SM-T870

samsung-devices-logs.zip

Google Pixel

Device Name	Android Version	Status
Pixel 9 Pro XL	15.0	✅ Success
Pixel 9 Pro	15.0	✅ Success
Pixel 9	15.0	✅ Success
Pixel 6 Pro	15.0, 13.0, 12.0	✅ Success
Pixel 8 Pro	14.0	✅ Success
Pixel 8	14.0	✅ Success
Pixel 7 Pro	13.0	✅ Success
Pixel 7	13.0	✅ Success
Pixel 6	12.0	✅ Success
Pixel 5	12.0, 11.0	❌ Failure

OnePlus

Device Name	Android Version	Status
12R	14.0	✅ Success
11R	13.0	✅ Success
9	11.0	✅ Success

Huawei

Device Name	Android Version	Status
Nova 11 SE	12.0	❌ Failure

Xiaomi

Device Name	Android Version	Status
Redmi Note 11	11.0	❌ Failure

Vivo

Device Name	Android Version	Status
Y21	11.0	❌ Failure
V21	11.0	❌ Failure

Oppo

Device Name	Android Version	Status
Reno 6	11.0	❌ Failure
A96	11.0	❌ Failure

Motorola

Device Name	Android Version	Status
Moto G71 5G	11.0	❌ Failure

Raw result: browser-stack-app-automate-tests.json

[freedomtan]

@anhappdev please help collected logs of failed Samsung devices and share them, so that we can inspect them.

[anhappdev]

Please refer to the updated comment: #1010 (comment). It includes new columns for SOC and model code. The log files are also available for download.

[freedomtan]

aswin: please help check failed qualcomm ones (see the logs prepared by @anhappdev above).
@Mostelk: please help check Exynos ones.

[Aswinoss]

@anhappdev From going through S24 ultra logs shared in the zip, we can see all tests passed and not able to identify any failure

Are these logs the same as the runs reported as failure, by Browser stack ?

[anhappdev]

@Aswinoss The log is correct. They are from here: https://github.com/mlcommons/mobile_app_open/actions/runs/15590751655?pr=1012
I just checked again. It must be my mistake to mark S24 Ultra run as Failure. I will update the table.

[Mostelk]

@swasson488 we have the list of OS: Android 11, 12, 13, 14, 15, list of SOCs keep it as release notes,
The app has tested on these phones ... @freedomtan @Mostelk @mohitmundhragithub please append list of phones which are also verified, please add here

[freedomtan]

@Aswinoss please help check other Qualcomm SoC-based Samsung devices.
@AhmedTElthakeb plase help check failed Exynos devices.

[Aswinoss]

@Aswinoss please help check other Qualcomm SoC-based Samsung devices. @AhmedTElthakeb plase help check failed Exynos devices.

@anhappdev I can see Galaxy tab S8 failed in multiple tests due to the output numbers being not in the approved range . These failures are only seen in that device.

Could you please tell, based on how these ranges are hard-coded and checked for ?

The other failures are all due to same reason, which we will have a fix soon, to avoid the crash

[anhappdev]

@Aswinoss You can find the range defined here:
https://github.com/mlcommons/mobile_app_open/blob/master/flutter/integration_test/expected_accuracy.dart

[Aswinoss]

@Aswinoss You can find the range defined here: https://github.com/mlcommons/mobile_app_open/blob/master/flutter/integration_test/expected_accuracy.dart

@anhappdev Could you please re run the tests for Galaxy tab s8. Cause I see there is no reason why the accuracy tests passed for other devices but not this.
Also, could we get the inputs executed and the corresponding outputs generated, if possible

[anhappdev]

@Aswinoss To print the inputs and outputs you will need to modify the backend.
I reran the test (https://github.com/mlcommons/mobile_app_open/actions/runs/15590751655/job/45056228807) and here is the log logs--Samsung Galaxy Tab S8-12.0-479.zip
By the way, the error isn't due to the accuracy range.