corelib.status unicode_test : Fail no longer works in 3.7.0

[eseidel]

Shorebird runs the Dart VM tests on our bots. We have to make a few changes to the .status files to get them to all pass cleanly (since upstream/Google doesn't seem to use status files exclusively anymore rather in combination with some expectation database).

However something changed in the handling of status files between Dart 3.6.2 and Dart 3.7.0 (Shorebird forks track stable). (Maybe it was earlier than that, but I think that's the revision.)

I replicated the behavior on stock 3.7.0 just now.

Repro instructions:

git checkout 3.7.0
gclient sync -D
./tools/build.py --no-rbe --mode debug --arch arm64 default dartaotruntime   

Confirm the test fails as expected (if not, make it fail by editing it, for me Straße isn't encoded correctly):

python3 tools/test.py -m debug -c dartkp -r dart_precompiled -a arm64 --use-elf  corelib/unicode_test

(Optional) delete all files in tests/corelib except unicode_test.dart and corelib.status to make your life easier (otherwise the tests will take several minutes to run).

Apply this diff to corelib.status:

diff --git a/tests/corelib/corelib.status b/tests/corelib/corelib.status
index 4ff77260d00..9aaaf97c5de 100644
--- a/tests/corelib/corelib.status
+++ b/tests/corelib/corelib.status
@@ -2,6 +2,8 @@
 # for details. All rights reserved. Use of this source code is governed by a
 # BSD-style license that can be found in the LICENSE file.
 
+unicode_test: Fail
+
 bigint_parse_radix_test/3: Slow # Issue http://dartbug.com/47050
 bigint_test: Pass, Slow 

That diff used to work to mark corelib/unicode_test.dart as failing for all configurations. But doesn't seem to any more as of 3.7.0

Now run "all" of corelib (now that it's only 1 test it will be very quick and see that unicode_test.dart still fails even though it's being marked as "Fail" in the .status file:

python3 tools/test.py -m debug -c dartkp -r dart_precompiled -a arm64 --use-elf  corelib
No build targets found.
Test configuration:
    custom-configuration-1(architecture: arm64, compiler: dartkp, mode: debug, runtime: dart_precompiled, system: mac)
Suites tested: corelib

FAILED: dartkp-dart_precompiled debug_arm64 corelib/unicode_test
Expected: Pass
Actual: RuntimeError

--- Command "vm_compile_to_kernel" (took 04.000674s):
DART_CONFIGURATION=DebugARM64 /Users/eseidel/Documents/GitHub/dart-sdk/sdk/pkg/vm/tool/gen_kernel --aot --platform=xcodebuild/DebugARM64/vm_platform_strong.dill -o /Users/eseidel/Documents/GitHub/dart-sdk/sdk/xcodebuild/DebugARM64/generated_compilations/custom-configuration-1/tests_corelib_unicode_test/out.dill /Users/eseidel/Documents/GitHub/dart-sdk/sdk/tests/corelib/unicode_test.dart -Dtest_runner.configuration=custom-configuration-1 --packages=/Users/eseidel/Documents/GitHub/dart-sdk/sdk/.dart_tool/package_config.json -Ddart.vm.product=false --sound-null-safety

exit code:
0

--- Command "precompiler" (took 03.000269s):
DART_CONFIGURATION=DebugARM64 xcodebuild/DebugARM64/gen_snapshot --snapshot-kind=app-aot-elf --elf=/Users/eseidel/Documents/GitHub/dart-sdk/sdk/xcodebuild/DebugARM64/generated_compilations/custom-configuration-1/tests_corelib_unicode_test/out.aotsnapshot --loading-unit-manifest=/Users/eseidel/Documents/GitHub/dart-sdk/sdk/xcodebuild/DebugARM64/generated_compilations/custom-configuration-1/tests_corelib_unicode_test/ignored.json --sound-null-safety -Dtest_runner.configuration=custom-configuration-1 --ignore-unrecognized-flags --packages=/Users/eseidel/Documents/GitHub/dart-sdk/sdk/.dart_tool/package_config.json /Users/eseidel/Documents/GitHub/dart-sdk/sdk/xcodebuild/DebugARM64/generated_compilations/custom-configuration-1/tests_corelib_unicode_test/out.dill

exit code:
0

--- Command "remove_kernel_file" (took 6ms):
DART_CONFIGURATION=DebugARM64 rm /Users/eseidel/Documents/GitHub/dart-sdk/sdk/xcodebuild/DebugARM64/generated_compilations/custom-configuration-1/tests_corelib_unicode_test/out.dill

exit code:
0

--- Command "vm" (took 23ms):
DART_CONFIGURATION=DebugARM64 TEST_COMPILATION_DIR=/Users/eseidel/Documents/GitHub/dart-sdk/sdk/xcodebuild/DebugARM64/generated_compilations/custom-configuration-1/tests_corelib_unicode_test xcodebuild/DebugARM64/dartaotruntime --sound-null-safety -Dtest_runner.configuration=custom-configuration-1 --ignore-unrecognized-flags --packages=/Users/eseidel/Documents/GitHub/dart-sdk/sdk/.dart_tool/package_config.json /Users/eseidel/Documents/GitHub/dart-sdk/sdk/xcodebuild/DebugARM64/generated_compilations/custom-configuration-1/tests_corelib_unicode_test/out.aotsnapshot

exit code:
255

stderr:
Unhandled exception:
Expect.equals(expected: <STRASSE>, actual: <STRA\xDFE>) fails.
#0      Expect._fail (package:expect/expect.dart:887)
#1      Expect._failNotEqual (package:expect/expect.dart:171)
#2      Expect.equals (package:expect/expect.dart:157)
#3      UnicodeTest.testMain (file:///Users/eseidel/Documents/GitHub/dart-sdk/sdk/tests/corelib/unicode_test.dart:10)
#4      main (file:///Users/eseidel/Documents/GitHub/dart-sdk/sdk/tests/corelib/unicode_test.dart:15)
#5      _delayEntrypointInvocation.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:315)
#6      _RawReceivePort._handleMessage (dart:isolate-patch/isolate_patch.dart:194)

--- Re-run this test:
python3 tools/test.py -m debug -c dartkp -r dart_precompiled -a arm64 --use-elf corelib/unicode_test
[00:08 | 100% | +    0 | -    1]

=== 0 tests passed, 1 failed ===

What am I missing? Thanks!

[eseidel]

I haven't yet bisected (mostly because deleting all the tests at each revision is a slight pain).

[eseidel]

I expect that I'm missing something obvious and this is simply my user error. But I've yet to figure out what.

[eseidel]

Hmm, my test setup fails in 3.6.2 as well, so clearly I'm confused. @felangel do you recall when we started seeing so many failures? (I'm still not sure why I can't seem to mark unicode_test as failing).

To try on 3.6.2 requires a slightly different build command because the target name changed:

./tools/build.py --no-rbe --mode debug --arch arm64 dart_precompiled_runtime default && say "done"   

[mraleph]

Yeah, I think this has not been working for a while. You can mark things as Skip, Crash or Slow but Fail marker is not respected. I think this is probably by design by now.

cc @athomas

[athomas]

This is WAI: #56660 (comment)

[eseidel]

Would just love some instruction on how someone is to maintain a fork of Dart. :)

Previously we (manually) maintained forks of status files. I'm not sure how to match what Dart does upstream with the results db. I guess forks of Dart are also expected to run their own copy of:
https://github.com/dart-lang/dart_ci/tree/main/current_results?
Since status files can no longer be used to encode "outcomes" per the status files doc:
https://github.com/dart-lang/sdk/blob/main/docs/Status-files.md
Also looks like that doc needs an update since it mentions RuntimeError, etc.

[athomas]

Would just love some instruction on how someone is to maintain a fork of Dart. :)

Previously we (manually) maintained forks of status files. I'm not sure how to match what Dart does upstream with the results db. I guess forks of Dart are also expected to run their own copy of: https://github.com/dart-lang/dart_ci/tree/main/current_results? Since status files can no longer be used to encode "outcomes" per the status files doc: https://github.com/dart-lang/sdk/blob/main/docs/Status-files.md Also looks like that doc needs an update since it mentions RuntimeError, etc.

It's not something we explicitly support. tools/test.dart does something similar for local testing that might be re-usable in your context.

This would produce the results.json files that capture the current status of tests:

tools/test.py --named-configuration=<some config> --output-directory=${outDirectory.path} --clean-exit --silent-failures --write-results

And this would compare a results.json file to a previous one:

tools/bots/compare_results.dart
--human
--verbose
--changed
--failing
--passing
--logs=${outDirectory.path}/logs.json
${outDirectory.path}/previous.json
${outDirectory.path}/results.json

Then you either produce both json files in one run (running with and without a patched change) or archive the results.json somewhere (you could check them in if you want, making them almost like status files).

[eseidel]

I appreciate the response.

[eseidel]

eseidel@mac sdk % ./tools/test.dart 
Please specify either a configuration (-n) or a builder (-b)

Usage: test.dart -b [BUILDER] -n [CONFIGURATION] [OPTION]... [--]
                 [TEST.PY OPTION]... [SELECTOR]...

Run tests and compare with the results on the given builder. Either the -n or
the -b option, or both, must be used. Any options following -- and non-option
arguments will be forwarded to test.py invocations. The specified named
configuration's results will be downloaded from the specified builder. If only a
named configuration is specified, the results are downloaded from the
appropriate builders. If only a builder is specified, the default named
configuration is used if the builder only has a single named configuration.
Otherwise the available named configurations are listed.

See the documentation at https://goto.google.com/dart-status-file-free-workflow

Run test.dart --help to see all options.

I can't even view that doc. 😭