Failures new nnbd VM builders on simarm targets

[a-siva]

There are new test failures on [infra] Add additional nnbd VM builders

language/mixin_constructor_forwarding/const_constructor_test/issue38304 (crashes instead of RuntimeError)
language/mixin_constructor_forwarding/mixin_constructor_parameter_forwarding_test (crashes instead of RuntimeError)

standalone/http_launch_test/0
standalone/http_launch_test/1
standalone/http_launch_test/2
standalone/http_launch_test/3

[alexmarkov]

language/mixin_constructor_forwarding/const_constructor_test and language/mixin_constructor_forwarding/mixin_constructor_parameter_forwarding_test are currently passing (https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket.appspot.com/8867760015291470256/+/steps/test_results/0/logs/tests_that_began_passing/0).

The fix for standalone/http_launch_test failures: https://dart-review.googlesource.com/c/sdk/+/165406

[a-siva]

We have some tests timing out

co19/LibTest/collection/ListMixin/ListMixin_class_A01_t05
co19/LibTest/collection/ListBase/ListBase_class_A01_t05
co19/LibTest/collection/ListBase/ListBase_class_A01_t06
co19/LibTest/collection/ListMixin/ListMixin_class_A01_t04
co19/LibTest/collection/ListMixin/ListMixin_class_A01_t06

[mkustermann]

The tests

co19/LibTest/collection/ListBase/ListBase_class_A01_t03 Timeout (expected Pass)
co19/LibTest/collection/ListBase/ListBase_class_A01_t04 Timeout (expected Pass)
co19/LibTest/collection/ListMixin/ListMixin_class_A01_t03 Timeout (expected Pass)
co19/LibTest/core/List/List_all_t03 Timeout (expected Pass)
co19/LibTest/core/List/List_all_t04 Timeout (expected Pass)
co19/LibTest/core/List/List_all_t05 Timeout (expected Pass)
co19/LibTest/core/List/List_all_t06 Timeout (expected Pass)

are failing on configurations

dartkp-strong-linux-debug-x64
dartkp-weak-asserts-linux-debug-x64

[alexmarkov]

The tests in co19/LibTest/collection/ListMixin/, co19/LibTest/collection/ListBase/ and co19/LibTest/core/List/ mentioned above perform sorting of large lists specified by enormous list literals (they include co19/src/LibTest/core/List/sort_A01_t05.test.dart, co19/src/LibTest/core/List/sort_A01_t06.test.dart, co19/src/LibTest/core/List/sort_A01_t07.test.dart or co19/src/LibTest/core/List/sort_A01_t08.test.dart).

I saw 2 kinds of timeouts on those tests:

• Timeouts at run time on simulated architectures (-simarm, -simarm64). Simulator is inherently slower than native architectures.
• Timeouts in gen_snapshot on AOT/debug configurations (dartkp-*-debug-*). In this case it looks like we have too heavy assertions in debug mode which slow down compilation considerably.

I wasn't able to reproduce some of these timeouts locally (although running those tests takes considerable time).
For timeouts which I could reproduce, corresponding co19_2 legacy test also times out.

These timeouts seems to be unrelated to NNBD. We just don't currently run legacy co19_2 tests on these configurations.

I think we can do one of the following:

• Skip those tests on the simulated and aot/debug configurations. This would make bots faster but potentially reduce test coverage.
• Mark them as Pass, Slow. This may or may not work depending on the test, and would result in slower bots and higher probability of flaky timeouts on these tests.
• Ask co19 folks to cut down those tests.

I'm leaning towards skipping those tests if $arch == simarm || $arch == simarm64 || ($runtime == dart_precompiled && $mode == debug).

@a-siva @mkustermann What do you think?

[mkustermann]

@alexmarkov As a general rule, we should avoid skipping tests or polluting status files / approval database. So if we can reduce the runtime of those tests by making them do less work without sacrificing what the test is actually testing, I would suggest to file a co19 issue to make them faster.