[macOS] Add run release test in devicelab #100526

cbracken · 2022-03-22T03:40:29Z

Adds a test that invokes flutter run in release mode on macOS desktop,
waits for successful launch and the flutter command list, then sends the
'q' command to quit the running app.

This adds an integration test for #100504.

Issue: #100348 (fix)
Issue: #97978 (partial fix)
Issue: #97977 (partial fix)
Umbrella issue: #60113

Pre-launch Checklist

I read the Contributor Guide and followed the process outlined there for submitting PRs.
I read the Tree Hygiene wiki page, which explains my responsibilities.
I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
I signed the CLA.
I listed at least one issue that this PR fixes in the description above.
I updated/added relevant documentation (doc comments with ///).
I added new tests to check the change I am making, or this PR is test-exempt.
All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

flutter-dashboard · 2022-03-22T03:40:32Z

It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).

If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

Adds a test that invokes flutter run in release mode on macOS desktop, waits for successful launch and the flutter command list, then sends the 'q' command to quit the running app. This adds an integration test for flutter#100504. Issue: flutter#100348

cbracken · 2022-03-22T03:46:23Z

Requested a test exemption from @Hixie on Discord.
Exemption reason: This is a test :)

I've filed #100527 to track patching the bot to treat changes under dev/devicelab/bin/tasks as tests.

jmagman · 2022-03-22T03:50:37Z

dev/devicelab/bin/tasks/run_release_test_macos.dart

+/// Basic launch test for desktop operating systems.
+void main() {
+  task(() async {
+    deviceOperatingSystem = DeviceOperatingSystem.macos;


You don't have to do it in this PR but can you also replace this deviceIdOverride hack with your nicer DeviceOperatingSystem.macos fix?

flutter/dev/devicelab/bin/tasks/hot_mode_dev_cycle_macos_target__benchmark.dart

Line 9 in deacb8d

await task(createHotModeTest(deviceIdOverride: 'macos'));

Thanks for spotting this. Looks like we do the same in the Windows/Linux equivalents too. I'll send one patch that does all three and adds the Linux DeviceOperatingSystem if it doesn't exist (the Windows one definitely does).

dev/devicelab/bin/tasks/run_release_test_macos.dart

jmagman

warning â�¢ Missing case clause for 'macos' â�¢ dev/devicelab/lib/tasks/perf_tests.dart:585:7 â�¢ missing_enum_constant_in_switch
warning â�¢ Missing case clause for 'macos' â�¢ dev/devicelab/lib/tasks/perf_tests.dart:705:7 â�¢ missing_enum_constant_in_switch
warning â�¢ Missing case clause for 'macos' â�¢ dev/devicelab/lib/tasks/perf_tests.dart:1299:5 â�¢ missing_enum_constant_in_switch
  error â�¢ The non-nullable local variable 'releaseSizeInBytes' must be assigned before it can be used â�¢ dev/devicelab/lib/tasks/perf_tests.dart:1363:29 â�¢ not_assigned_potentially_non_nullable_local_variable
warning â�¢ Missing case clause for 'macos' â�¢ dev/devicelab/lib/tasks/perf_tests.dart:1378:5 â�¢ missing_enum_constant_in_switch

keyonghan

Config LGTM with nits.

.ci.yaml

TESTOWNERS

Hixie · 2022-03-22T06:34:10Z

test-exempt: is a test

dev/devicelab/bin/tasks/run_release_test_macos.dart

jmagman

LGTM, thanks!

…0526)

jmagman · 2022-03-22T22:16:35Z

This is failing:
https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8818956881147077521/+/u/run_run_release_test_macos/test_stdout

macOS builds spew so much to stderr...

In the run_release_test_macos integration test that verifies that a release build of an app can be launched (and quit), xcodebuild from the Xcode install on the macOS bots emits a few info messages about Simulator SDK versions that are irrelevant to the functioning of this test. Ignore these instead of failing the test if they occur. Related: flutter#100526 Issue: flutter#100348 (fix) Issue: flutter#97978 (partial fix) Issue: flutter#97977 (partial fix) Umbrella issue: flutter#60113

In the run_release_test_macos integration test that verifies that a release build of an app can be launched (and quit), xcodebuild from the Xcode install on the macOS bots emits a few info messages about Simulator SDK versions that are irrelevant to the functioning of this test. Ignore these instead of failing the test if they occur. Related: #100526 Issue: #100348 (fix) Issue: #97978 (partial fix) Issue: #97977 (partial fix) Umbrella issue: #60113

cbracken · 2022-03-23T01:44:30Z

Workaround for the failure landed in #100591.

…0526)

cbracken requested review from jmagman and zanderso March 22, 2022 03:40

cbracken requested review from keyonghan and CaseyHillers as code owners March 22, 2022 03:40

flutter-dashboard bot added the team Infra upgrades, team productivity, code health, technical debt. See also team: labels. label Mar 22, 2022

This was referenced Mar 22, 2022

[macOS] Use arm64 snapshot in arm64 App.framework #100504

Merged

[macOS][m1] App crashes in release mode upon app launch #100348

Closed

cbracken added a: tests "flutter test", flutter_test, or one of our tests platform-mac Building on or for macOS specifically a: desktop Running on desktop c: tech-debt Technical debt, code quality, testing, etc. labels Mar 22, 2022

cbracken force-pushed the add-run-release-macos branch from 7271819 to 16e4fae Compare March 22, 2022 03:44

cbracken mentioned this pull request Mar 22, 2022

Test exemption bot shouldn't flag for missing tests if devicelab tests are changed #100527

Closed

jmagman reviewed Mar 22, 2022

View reviewed changes

keyonghan approved these changes Mar 22, 2022

View reviewed changes

.ci.yaml Outdated Show resolved Hide resolved

TESTOWNERS Outdated Show resolved Hide resolved

cbracken added 3 commits March 21, 2022 21:19

Use startFlutter instead of startProcess

dcb68e4

Add missing cases

ea480b7

Indentation, move test to host-only devicelab sec.

87bc977

keyonghan mentioned this pull request Mar 22, 2022

Add Mac M1 ARM machines to the prod pool #87508

Closed

jmagman reviewed Mar 22, 2022

View reviewed changes

dev/devicelab/bin/tasks/run_release_test_macos.dart Outdated Show resolved Hide resolved

dev/devicelab/bin/tasks/run_release_test_macos.dart Show resolved Hide resolved

cbracken added 2 commits March 22, 2022 11:53

Add isBot param to startFlutter

74dac2c

Use hello world example

dd3bf73

jmagman approved these changes Mar 22, 2022

View reviewed changes

cbracken merged commit 4b81978 into flutter:master Mar 22, 2022

cbracken deleted the add-run-release-macos branch March 22, 2022 21:13

engine-flutter-autoroll mentioned this pull request Mar 22, 2022

Roll Flutter from 8e7b3616513f to ea8aeb7427f4 (2 revisions) flutter/packages#1316

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 22, 2022

4b81978 [macOS] Add run release test in devicelab (flutter/flutter#10…

038cfc4

…0526)

cbracken mentioned this pull request Mar 22, 2022

[macOS] Ignore stderr contents in launch test #100591

Merged

8 tasks

engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Mar 29, 2022

4b81978 [macOS] Add run release test in devicelab (flutter/flutter#10…

306df4b

…0526)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Mar 29, 2022

4b81978 [macOS] Add run release test in devicelab (flutter/flutter#10…

6e52d67

…0526)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Mar 29, 2022

4b81978 [macOS] Add run release test in devicelab (flutter/flutter#10…

272fc06

…0526)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Mar 29, 2022

4b81978 [macOS] Add run release test in devicelab (flutter/flutter#10…

9f55991

…0526)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Mar 29, 2022

4b81978 [macOS] Add run release test in devicelab (flutter/flutter#10…

e77ba3f

…0526)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Mar 29, 2022

4b81978 [macOS] Add run release test in devicelab (flutter/flutter#10…

215f1c9

…0526)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Mar 29, 2022

4b81978 [macOS] Add run release test in devicelab (flutter/flutter#10…

23232c2

…0526)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[macOS] Add run release test in devicelab #100526

[macOS] Add run release test in devicelab #100526

cbracken commented Mar 22, 2022 •

edited

Loading

flutter-dashboard bot commented Mar 22, 2022

cbracken commented Mar 22, 2022 •

edited

Loading

jmagman Mar 22, 2022

cbracken Mar 22, 2022

jmagman left a comment

keyonghan left a comment

Hixie commented Mar 22, 2022

jmagman left a comment

jmagman commented Mar 22, 2022

cbracken commented Mar 23, 2022

[macOS] Add run release test in devicelab #100526

[macOS] Add run release test in devicelab #100526

Conversation

cbracken commented Mar 22, 2022 • edited Loading

Pre-launch Checklist

flutter-dashboard bot commented Mar 22, 2022

cbracken commented Mar 22, 2022 • edited Loading

jmagman Mar 22, 2022

Choose a reason for hiding this comment

cbracken Mar 22, 2022

Choose a reason for hiding this comment

jmagman left a comment

Choose a reason for hiding this comment

keyonghan left a comment

Choose a reason for hiding this comment

Hixie commented Mar 22, 2022

jmagman left a comment

Choose a reason for hiding this comment

jmagman commented Mar 22, 2022

cbracken commented Mar 23, 2022

cbracken commented Mar 22, 2022 •

edited

Loading

cbracken commented Mar 22, 2022 •

edited

Loading