Fix @failingTest not working correctly for out-of-band exceptions

[DanTup]

The _runFailingTest method is documented with:

- An exception is thrown to the zone handler from a timer task.

However, if the test code runs without exceptions, it would still call test_package.fail('Test passed - expected to fail.'); (and cause the test to not complete until the timeout).

This means you could have a test (such as the one added here) that fails when run without @failingTest but hands and reports Test passed when run with it.

Fixes Dart-Code/Dart-Code#5761

Before:

After:

---

• I’ve reviewed the contributor guide and applied the relevant portions to this PR.

[gemini-code-assist]

Code Review

This pull request correctly fixes an issue where @failingTest did not handle out-of-band exceptions properly, which could lead to tests hanging. The introduction of a failed flag effectively tracks whether any exception has occurred, preventing the test from being incorrectly marked as 'passed' and then failed. The new test case test_fails_throws_outOfBand is a great addition that specifically targets and verifies the fix.

I've added a couple of suggestions to improve the code further:

• Refactoring duplicated error-handling logic in _runFailingTest to improve maintainability.
• Aligning the new test function's signature with Effective Dart guidelines for better type safety, as per the repository's style guide.

[scheglov]

The problem is real, waiting for these timeouts is very inconvenient.

I'm not sure it requires any fancy out-of-band code.
Or maybe there are more than one issue :-)

This code:

  @failingTest
  void test_xxx() {}

causes

00:00 +0 -1: ClassTypeAliasResolutionTest | test_xxx [E]                                                                                                                                               
  Test passed - expected to fail.
  package:matcher/src/expect/expect.dart 187:31                      fail
  package:test_reflective_loader/test_reflective_loader.dart 255:20  _runFailingTest.<fn>.<fn>
  dart:async/zone.dart 1538:47                                       _rootRunUnary
  dart:async/zone.dart 1429:19                                       _CustomZone.runUnary

<30 seconds wait>
00:30 +8 -1: Some tests failed. 

The final statistics is correct.
The timeout - not.

Looking at your picture again... Well, maybe we do have more than one issue here.
I wonder if they are related and both can be fixed with one change.

[DanTup]

Looking at your picture again... Well, maybe we do have more than one issue here.

If you mean the second failing test (the teardown one), that was just because that test doesn't work correctly if you use @soloTest (because it doesn't run the second test class). I didn't realise that until after taking the screenshot 😄

However, looks like you're right there is still an issue with it taking a long time in your repro - I will have a look at that.

[DanTup]

I forgot to mention in my comment above - I think the reason for the timeout was this:

https://api.dart.dev/dart-async/runZonedGuarded.html

The created zone will always be an error zone. Asynchronous errors in futures never cross zone boundaries between zones with a different Zone.errorZone. A consequence of that behavior can be that a Future which completes as an error in the created zone will seem to never complete when used from a zone that belongs to a different error zone.

So I changed it to not throw inside the callback but instead throw it outside (although I'm not sure I completely understand why other exceptions thrown by tests didn't lead to the same condition).

[DanTup]

@scheglov thanks! I don't have any permissions here - are you the right person to approve the bots and/or merge, or does it need someone else?

[github-actions]

Package publishing

Package	Version	Status	Publish tag (post-merge)
package:bazel_worker	1.1.4	already published at pub.dev
package:benchmark_harness	2.4.0-wip	WIP (no publish necessary)
package:boolean_selector	2.1.2	already published at pub.dev
package:browser_launcher	1.1.3	already published at pub.dev
package:cli_config	0.2.1-wip	WIP (no publish necessary)
package:cli_util	0.5.0-wip	WIP (no publish necessary)
package:clock	1.1.3-wip	WIP (no publish necessary)
package:code_builder	4.11.0	already published at pub.dev
package:coverage	1.15.0	already published at pub.dev
package:csslib	1.0.2	already published at pub.dev
package:extension_discovery	2.1.0	already published at pub.dev
package:file	7.0.2-wip	WIP (no publish necessary)
package:file_testing	3.1.0-wip	WIP (no publish necessary)
package:glob	2.1.3	already published at pub.dev
package:graphs	2.3.3-wip	WIP (no publish necessary)
package:html	0.15.7-wip	WIP (no publish necessary)
package:io	1.1.0-wip	WIP (no publish necessary)
package:json_rpc_2	4.0.0	already published at pub.dev
package:markdown	7.3.1-wip	WIP (no publish necessary)
package:mime	2.0.0	already published at pub.dev
package:oauth2	2.0.5	already published at pub.dev
package:package_config	2.3.0-wip	WIP (no publish necessary)
package:pool	1.5.2	already published at pub.dev
package:process	5.0.5	already published at pub.dev
package:pub_semver	2.2.0	already published at pub.dev
package:pubspec_parse	1.5.1-wip	WIP (no publish necessary)
package:source_map_stack_trace	2.1.3-wip	WIP (no publish necessary)
package:source_maps	0.10.14-wip	WIP (no publish necessary)
package:source_span	1.10.1	already published at pub.dev
package:sse	4.1.8	already published at pub.dev
package:stack_trace	1.12.1	already published at pub.dev
package:stream_channel	2.1.4	already published at pub.dev
package:stream_transform	2.1.2-wip	WIP (no publish necessary)
package:string_scanner	1.4.1	already published at pub.dev
package:term_glyph	1.2.3-wip	WIP (no publish necessary)
package:test_reflective_loader	0.5.0	ready to publish	test_reflective_loader-v0.5.0
package:timing	1.0.2	already published at pub.dev
package:unified_analytics	8.0.6	ready to publish	unified_analytics-v8.0.6
package:watcher	1.1.5-wip	WIP (no publish necessary)
package:yaml	3.1.3	already published at pub.dev
package:yaml_edit	2.2.2	already published at pub.dev

Documentation at https://github.com/dart-lang/ecosystem/wiki/Publishing-automation.

[github-actions]

PR Health

Breaking changes ✔️

Package	Change	Current Version	New Version	Needed Version	Looking good?
test_reflective_loader	None	0.4.0	0.5.0	0.4.0	✔️

This check can be disabled by tagging the PR with skip-breaking-check.

Changelog Entry ✔️

Package	Changed Files

Changes to files need to be accounted for in their respective changelogs.

This check can be disabled by tagging the PR with skip-changelog-check.

Coverage ✔️

File	Coverage
pkgs/test_reflective_loader/lib/test_reflective_loader.dart	💚 87 % ⬆️ 1 %

This check for test coverage is informational (issues shown here will not fail the PR).

This check can be disabled by tagging the PR with skip-coverage-check.

API leaks ✔️

The following packages contain symbols visible in the public API, but not exported by the library. Export these symbols or remove them from your publicly visible API.

Package	Leaked API symbol	Leaking sources

This check can be disabled by tagging the PR with skip-leaking-check.

License Headers ✔️

// Copyright (c) 2025, the Dart project authors. Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.

Files
no missing headers

All source files should start with a license header.

Unrelated files missing license headers

Files
pkgs/bazel_worker/benchmark/benchmark.dart
pkgs/benchmark_harness/integration_test/perf_benchmark_test.dart
pkgs/boolean_selector/example/example.dart
pkgs/clock/lib/clock.dart
pkgs/clock/lib/src/clock.dart
pkgs/clock/lib/src/default.dart
pkgs/clock/lib/src/stopwatch.dart
pkgs/clock/lib/src/utils.dart
pkgs/clock/test/clock_test.dart
pkgs/clock/test/default_test.dart
pkgs/clock/test/stopwatch_test.dart
pkgs/clock/test/utils.dart
pkgs/coverage/lib/src/coverage_options.dart
pkgs/html/example/main.dart
pkgs/html/lib/dom.dart
pkgs/html/lib/dom_parsing.dart
pkgs/html/lib/html_escape.dart
pkgs/html/lib/parser.dart
pkgs/html/lib/src/constants.dart
pkgs/html/lib/src/encoding_parser.dart
pkgs/html/lib/src/html_input_stream.dart
pkgs/html/lib/src/list_proxy.dart
pkgs/html/lib/src/query_selector.dart
pkgs/html/lib/src/token.dart
pkgs/html/lib/src/tokenizer.dart
pkgs/html/lib/src/treebuilder.dart
pkgs/html/lib/src/utils.dart
pkgs/html/test/dom_test.dart
pkgs/html/test/parser_feature_test.dart
pkgs/html/test/parser_test.dart
pkgs/html/test/query_selector_test.dart
pkgs/html/test/selectors/level1_baseline_test.dart
pkgs/html/test/selectors/level1_lib.dart
pkgs/html/test/selectors/selectors.dart
pkgs/html/test/support.dart
pkgs/html/test/tokenizer_test.dart
pkgs/html/test/trie_test.dart
pkgs/html/tool/generate_trie.dart
pkgs/pubspec_parse/test/git_uri_test.dart
pkgs/stack_trace/example/example.dart
pkgs/watcher/test/custom_watcher_factory_test.dart
pkgs/yaml_edit/example/example.dart

This check can be disabled by tagging the PR with skip-license-check.

[scheglov]

I can land it.
But it looks that we need to update pubspec and change log.

[DanTup]

@scheglov thanks! I've updated the version and changelog now.