Migrate Linux Framework Smoke Tests to engine v2

[godofredoc]

https://ci.chromium.org/p/flutter/builders/prod/Linux%20Framework%20Smoke%20Tests/12035

[keyonghan]

Created initial config file: https://github.com/flutter/engine/blob/b8fc36086e55038a4b0df8a1d6cde91e4dfb3e00/ci/builders/linux_framework_smoke.json

Led shows successful build, but test fails when downloading flutter dependencies: https://luci-milo.appspot.com/raw/build/logs.chromium.org/flutter/led/keyonghan_google.com/25126056de267e98e032dd46ba02567db559a2613be287f73e4642c0f248e8b3/+/build.proto

Not sure if this suggests engine was not build correctly.

[godofredoc]

It is expecting this build config https://cs.opensource.google/flutter/engine/+/master:ci/builders/linux_host_engine.json;l=67?q=dart-sdk-linux-x64.zip&ss=flutter%2Fengine:ci%2Fbuilders%2F along with the archives.

[keyonghan]

With the Archive and corresponding ninja targets, the build seems fine: https://github.com/flutter/engine/blob/442f5b768e0b52474678e501adb049b5fe061e0f/ci/builders/linux_framework_smoke.json

But the tests are consistently failing due to warning logs: https://ci.chromium.org/raw/build/logs.chromium.org/flutter/led/keyonghan_google.com/bd9aad521c426c5dc001e9df2227d748e13f2be6fe510cb447e4378b6531b51b/+/build.proto?server=chromium-swarm.appspot.com

00:15 +1: All tests passed!                                                                                                                                                                            
Logger received warning output during the run, and "--fatal-warnings" is enabled.
╔═╡ERROR╞═══════════════════════════════════════════════════════════════════════
║ Command: ../../bin/flutter test --test-randomize-ordering-seed=20230602 --fatal-warnings --dart-define=dart.vm.product=false --dart-define=dart.vm.profile=true test_profile/
║ Command exited with exit code 1 but expected zero exit code.
║ Working directory: /b/s/w/ir/cache/flutter/packages/flutter
║ stdout and stderr output:
║ (stdout/stderr output was more than 10 lines)
╚═══════════════════════════════════════════════════════════════════════════════
▌12:04:29▐ Test failed.

[godofredoc]

With the Archive and corresponding ninja targets, the build seems fine: https://github.com/flutter/engine/blob/442f5b768e0b52474678e501adb049b5fe061e0f/ci/builders/linux_framework_smoke.json

But the tests are consistently failing due to warning logs: https://ci.chromium.org/raw/build/logs.chromium.org/flutter/led/keyonghan_google.com/bd9aad521c426c5dc001e9df2227d748e13f2be6fe510cb447e4378b6531b51b/+/build.proto?server=chromium-swarm.appspot.com

00:15 +1: All tests passed!                                                                                                                                                                            
Logger received warning output during the run, and "--fatal-warnings" is enabled.
╔═╡ERROR╞═══════════════════════════════════════════════════════════════════════
║ Command: ../../bin/flutter test --test-randomize-ordering-seed=20230602 --fatal-warnings --dart-define=dart.vm.product=false --dart-define=dart.vm.profile=true test_profile/
║ Command exited with exit code 1 but expected zero exit code.
║ Working directory: /b/s/w/ir/cache/flutter/packages/flutter
║ stdout and stderr output:
║ (stdout/stderr output was more than 10 lines)
╚═══════════════════════════════════════════════════════════════════════════════
▌12:04:29▐ Test failed.

Nice, this is something we need to fix if this will be the new way of running framework tests in the engine. @christopherfujino what is the recommended way to fix this? it is also impacting dart in the same way.

@whesse FYI

[christopherfujino]

Hmm, this is pretty difficult to read: https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8779376220126520753/+/u/Run_framework_tests_libraries_tests/stdout

@gspencergoog am I missing something here, or is the test runner failing because of unexpected warning output but not actually showing the warning?

[whesse]

The warning causing the fatal exit is the 5th line of the log, saying "artifacts will be downloaded from ..."

I've been thinking about this, and propose adding a flag that would make the custom url warning not a fatal warning. That is the least intrusive. Or maybe the custom artifact url could always be non fatal when fatal warnings is enabled.

[christopherfujino]

The warning causing the fatal exit is the 5th line of the log, saying "artifacts will be downloaded from ..."

Ahh yeah, of course.

[christopherfujino]

I've been thinking about this, and propose adding a flag that would make the custom url warning not a fatal warning. That is the least intrusive. Or maybe the custom artifact url could always be non fatal when fatal warnings is enabled.

In taking a quick look at the code, I suspect the former would be easier.

[whesse]

Issue #116899 is already filed about this issue as well. I was planning to fix it next week, but need consensus on a solution.

[whesse]

I have created two PRs, one of which adds an option to suppress this fatal warning (but keep it a non-fatal warning), and the other to unconditionally make it a non-fatal warning. Note there is already a warning in the cache implementation that is non-fatal even when the --fatal-warnings flag is given.

[keyonghan]

The test config is ready to go (except the warning).
Here is the config: flutter/engine#42467, which uses three existing targets to cover the framework smoke tests:

• framework_tests_widget shard
• framework_tests_libraries shard
• analyze adhoc validation test

Corresponding recipes support: https://flutter-review.googlesource.com/c/recipes/+/45400

After the warning issue is addressed, we can enable the target in our CI.

[whesse]

The fatal warning due to the custom download URL should be gone now, my fix has landed. Led runs testing this are still pending.

[keyonghan]

The fatal warning due to the custom download URL should be gone now, my fix has landed. Led runs testing this are still pending.

Thanks @whesse for the fix! Validated tests are passing now: https://ci.chromium.org/raw/build/logs.chromium.org/flutter/led/keyonghan_google.com/5d35ba2b3f79764bfe58f10e4cd1bce118c3bd485ece5aff184183b232d181da/+/build.proto?server=chromium-swarm.appspot.com.

So the only blocker here on this bug is make artifacts hermetic to the builder.

[flutter-triage-bot]

This issue is assigned to @keyonghan but has had no recent status updates. Please consider unassigning this issue if it is not going to be addressed in the near future. This allows people to have a clearer picture of what work is actually planned. Thanks!

[flutter-triage-bot]

This issue was assigned to @keyonghan but has had no status updates in a long time. To remove any ambiguity about whether the issue is being worked on, the assignee was removed.

[keyonghan]

The target has been removed from Engine repo. Closing as obsolete.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.