Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Linux customer_testing is 2.00% flaky #149230

Closed
fluttergithubbot opened this issue May 29, 2024 · 6 comments · Fixed by flutter/tests#374
Closed

Linux customer_testing is 2.00% flaky #149230

fluttergithubbot opened this issue May 29, 2024 · 6 comments · Fixed by flutter/tests#374
Assignees
Labels
c: flake Tests that sometimes, but not always, incorrectly pass P0 Critical issues such as a build break or regression team-framework Owned by Framework team

Comments

@fluttergithubbot
Copy link
Contributor

The post-submit test builder Linux customer_testing had a flaky ratio 2.00% for the past (up to) 100 commits, which is above our 2.00% threshold.

One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20customer_testing/19403
Commit: 73fb6c7

Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20customer_testing/19403
https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20customer_testing/19364

Recent test runs:
https://flutter-dashboard.appspot.com/#/build?taskFilter=Linux%20customer_testing

Please follow https://github.com/flutter/flutter/wiki/Reducing-Test-Flakiness#fixing-flaky-tests to fix the flakiness and enable the test back after validating the fix (internal dashboard to validate: go/flutter_test_flakiness).

@fluttergithubbot fluttergithubbot added c: flake Tests that sometimes, but not always, incorrectly pass P0 Critical issues such as a build break or regression team-framework Owned by Framework team labels May 29, 2024
@goderbauer
Copy link
Member

One of these is the same analyzer crash as also seen on Mac: #149231 (comment)

@fluttergithubbot
Copy link
Contributor Author

[prod pool] flaky ratio for the past (up to) 100 commits between 2024-05-28 and 2024-06-04 is 1.00%. Flaky number: 1; total number: 100.
One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20customer_testing/19426
Commit: 35ae519
Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20customer_testing/19426

Recent test runs:
https://flutter-dashboard.appspot.com/#/build?taskFilter=Linux%20customer_testing

@goderbauer
Copy link
Member

The analyhzer crash seems to be gone. The new failure is a timeout in the zulip tests:

| 00:51 +1005 ~2 -1: /b/s/w/ir/x/t/flutter_customer_testing.zulip.EQJPYX/tests/test/model/store_test.dart: UpdateMachine.load retries registerQueue on NetworkError [E]
|   Bad state: Exceeded timeout 1:00:00.000000 while flushing timers
|   package:fake_async/fake_async.dart 217:9   FakeAsync.flushTimers.<fn>
|   package:fake_async/fake_async.dart 241:21  FakeAsync._fireTimersWhile
|   package:fake_async/fake_async.dart 214:5   FakeAsync.flushTimers
|   test/fake_async.dart 29:7                  awaitFakeAsync
|   test/model/store_test.dart 176:57          main.<fn>.<fn>

@goderbauer
Copy link
Member

cc @gnprice for zulip

@gnprice
Copy link
Member

gnprice commented Jun 5, 2024

Thanks. We've seen that flake a handful of times before:
zulip/zulip-flutter#602
but now that we're in flutter/tests we should really clean it up, because there's a lot more PRs per week in the Flutter repo than in Zulip. I'll go investigate, and either fix the test or disable it.

Then I believe that's the only known flaky test in the zulip-flutter repo. (Though I suppose if we have any really low-frequency flakes, then flutter/tests may be what eventually causes us to notice them!)

Copy link

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jun 20, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
c: flake Tests that sometimes, but not always, incorrectly pass P0 Critical issues such as a build break or regression team-framework Owned by Framework team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants