Collect devicelab android benchmarks to run on high-end phones

[keyonghan]

Flutter EngProd is working to support high-end phones like Samsung S10 and Pixel in devicelab. It would be useful to separate benchmarks between phones.

All benchmarks will run on low-end phones as of now, and this bug tracks what subset of benchmarks we want to also run on high-end phones.

Could you help identify ones we also want to run on high-end phones? @dnfield @gspencergoog @blasten @zanderso Please see the current list of android benchmarks on linux/mac/windows below.

List of Linux/Android benchmarks:

analyzer_benchmark
android_stack_size_test
android_view_scroll_perf__timeline_summary
animated_image_gc_perf
animated_placeholder_perf__e2e_summary
backdrop_filter_perf__e2e_summary
backdrop_filter_perf__timeline_summary
basic_material_app_android__compile
color_filter_and_fade_perf__e2e_summary
complex_layout_android__compile
complex_layout_android__scroll_smoothness
complex_layout_scroll_perf__devtools_memory
complex_layout_semantics_perf
complex_layout__start_up
complex_layout_scroll_perf__memory
complex_layout_scroll_perf__timeline_summary
cubic_bezier_perf__e2e_summary
cubic_bezier_perf_sksl_warmup__e2e_summary
cubic_bezier_perf_sksl_warmup__timeline_summary
cubic_bezier_perf__timeline_summary
cull_opacity_perf__e2e_summarycull_opacity_perf__timeline_summary
fading_child_animation_perf__timeline_summary
fast_scroll_large_images__memory
fast_scroll_heavy_gridview__memory
flutter_engine_group_performance
flutter_gallery__back_button_memory
flutter_gallery__image_cache_memory
flutter_gallery__memory_nav
flutter_gallery__start_up
flutter_gallery__start_up_delayed
flutter_gallery__transition_perf_e2e
flutter_gallery__transition_perf_hybrid
flutter_gallery__transition_perf
flutter_gallery__transition_perf_with_semantics
flutter_gallery_android__compile
flutter_gallery_sksl_warmup__transition_perf_e2e
flutter_gallery_sksl_warmup__transition_perf
flutter_gallery_v2_web_compile_test
flutter_test_performance
frame_policy_delay_test_android
fullscreen_textfield_perf__timeline_summary
hello_world__memory
hello_world_android__compile
home_scroll_perf__timeline_summary
hot_mode_dev_cycle__benchmark
hot_mode_dev_cycle_linux__benchmark
image_list_jit_reported_duration
image_list_reported_duration
imagefiltered_transform_animation_perf__timeline_summary
large_image_changer_perf_android
linux_chrome_dev_mode
microbenchmarks
multi_widget_construction_perf__e2e_summary
new_gallery__crane_perf
new_gallery__transition_perf
picture_cache_perf__timeline_summary
picture_cache_perf__e2e_summary
platform_channels_benchmarks
platform_views_scroll_perf__timeline_summary
textfield_perf__e2e_summary
textfield_perf__timeline_summary
tiles_scroll_perf__timeline_summary
web_size__compile_test

List of Mac/Android benchmarks:

flutter_gallery_mac__start_up

List of Windows/Android benchmarks:

basic_material_app_win__compile
complex_layout_win__compile
flutter_gallery_win__compile
hot_mode_dev_cycle_win__benchmark
windows_chrome_dev_mode

Related issue to move mac/android benchmarks to linux/android: #74522

/cc @godofredoc

[godofredoc]

An additional data point is that we are only planning to enable 3 test beds with these phones.

[zanderso]

An additional data point is that we are only planning to enable 3 test beds with these phones.

So how many benchmarks do you think we should aim for to avoid regressing cycle times? 5? 10? 25?

[keyonghan]

An additional data point is that we are only planning to enable 3 test beds with these phones.

So how many benchmarks do you think we should aim for to avoid regressing cycle times? 5? 10? 25?

We expect to have ~15 test beds with old moto g4 phones EOQ. Considering 3 test beds with high-end phones, ~10 benchmarks sound a reasonable number to run. (they will still run on moto g4 test beds btw).

[keyonghan]

@zanderso A quick follow up here. If we target for 10 benchmarks on high-end phones, do you have a list in mind? I am afraid we will target only for linux/android due to limited capacity.

[zanderso]

In no particular order:

1. backdrop_filter_perf__timeline_summary
2. complex_layout_scroll_perf__timeline_summary
3. cubic_bezier_perf__timeline_summary
4. cull_opacity_perf__timeline_summary
5. flutter_gallery__transition_perf
6. imagefiltered_transform_animation_perf__timeline_summary
7. new_gallery__transition_perf
8. picture_cache_perf__timeline_summary
9. platform_views_scroll_perf__timeline_summary
10. textfield_perf__timeline_summary

@knopp @iskakaushik do you want to evict one of these in favor of a benchmark that will catch regressions in Android DRM on newer devices?

[keyonghan]

These tests have been passing in staging consistently: https://ci.chromium.org/p/flutter/g/devicelab_staging/console.
Skia perf is showing their metrics, but is missing the correct device type tags:

This is due to recipe side missing staging builders.

[keyonghan]

Before enabled in prod (migrated 2 test beds to prod + enable tests in prod), we need to have at least one more test bed in staging.
Expected: 2 linux/samsungS10 in prod, and 1 linux/samsungS10 in staging.

[keyonghan]

https://flutter-review.googlesource.com/c/infra/+/27445 to populate tags property for staging builders.
https://flutter-review.googlesource.com/c/recipes/+/27361 to collect devices info based on target's tags property to make it scalable.

[keyonghan]

Chrom config cl/4581132 to enable 1 linux/samsungS10 in staging.

[keyonghan]

cl/4584031 to migrate 2 linux/samsungS10 to prod.

[keyonghan]

Tests enabled in CI and are running on samsung s10 tests. Metrics are being collected: https://flutter-flutter-perf.skia.org/e/?queries=device_type%3DSM-G973U1%26test%3Dplatform_views_scroll_perf__timeline_summary

Closing. Will create a separate one to track if we plan to support on other devices.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.