CFE test timing out on Windows (was flaky)

[whesse]

There are new test failures on Abstract field support: do not allow initializing abstract fields..

The test
pkg/front_end/test/incremental_dart2js_test Timeout (expected Pass)

is failing on configuration

cfe-unittest-asserts-release-win

It has been timing out for 6 months flakily, and now is always timing out.

[johnniwinther]

@jensjoha

[jensjoha]

Interesting... It seems to flake on the linux bots as well (at least the one bot I checked seems to have it as well) --- locally the test takes ~1 minute. And a single compile that locally takes ~5 seconds on the bots take ~21 seconds.
Are the bots - on a single core - expected to be ~4 times slower than our workstations?

[whesse]

In general, yes, that seems about right, CI builders are much slower than local runs. Are these compiles multi-threaded in any way? I think more tests run concurrently on local stations with 32 cores, at least.

[jensjoha]

(First a little clear-up: It's "incremental_compiler_leak_test" --- right?)

It:

• Launches "incremental_dart2js_tester.dart" with observatory enabled
• Whenever it pauses, it uses the observatory features to create a list of allocations for specific classes. It then unpause the script
• Once the launches script exits it performs some calculations

So basically it only does 1 thing at a time, logically using only 1 core.

[whesse]

This issue is actually about incremental_dart2js_test, and that test is the one that is now considered stable, and disappeared from the go/dart-flakiness dashboard. The other one is still considered flaky, but you should fix both of them. Did you find the incremental_compiler_leak_test name from the dashboard?

I'm working on a change that would keep tests that stop being flaky visible on the flakiness dashboard, or maybe on a new expired-flakiness-dashboard, so you could see their flakiness history when approving the new stable failure.

The flaky.json file from that builder, on the build before the test was marked stable (100 runs after the last flake), contains 2 records, one for each test:

{"configuration":"cfe-unittest-asserts-release-win","name":"pkg/front_end/test/incremental_compiler_leak_test","expected":"Pass","outcomes":["DartkCrash","Fail","Pass","RuntimeError","Timeout"],"last_new_result_seen":"2020-06-16T17:05:52.547547","current":"Timeout","current_counter":3,"occurrences":{"Timeout":1391,"Pass":955,"RuntimeError":5,"Fail":1,"DartkCrash":1},"first_seen":{"Timeout":"2020-03-03T11:53:54.481662","Pass":"2020-03-03T11:53:54.481662","RuntimeError":"2020-03-05T16:16:38.132834","Fail":"2020-05-13T05:38:24.332233","DartkCrash":"2020-06-16T17:05:52.547547"},"last_seen":{"Timeout":"2020-08-07T23:22:24.226946","Pass":"2020-08-07T18:57:36.096148","RuntimeError":"2020-07-14T14:20:26.450407","Fail":"2020-05-13T05:38:24.332233","DartkCrash":"2020-06-16T17:05:52.547547"},"matches":{"Timeout":false,"Pass":true,"RuntimeError":false,"Fail":false,"DartkCrash":false},"build_ids":{"Timeout":"8872558816791850768","Pass":"8872575344039025808","RuntimeError":"8874767727843926880","Fail":"8880418633967351728","DartkCrash":"8877293833016252192"},"commits":{"Timeout":"594434d5a0b530cfac750ca681ad059fd90d76fc","Pass":"940d7effbdab047afd0122b5d9bab6fdbbf48dd7","RuntimeError":"cfeb1d647ae59220747a56bec6d19db53c1f1e7e","Fail":"467ba2f3fb7231e746f463c52e77567759c91afe","DartkCrash":"29e93bcdbdf0c3081999aa247bd3251bf58edbcd"}}
{"configuration":"cfe-unittest-asserts-release-win","name":"pkg/front_end/test/incremental_dart2js_test","expected":"Pass","outcomes":["DartkCrash","Pass","Timeout"],"last_new_result_seen":"2020-06-16T17:05:52.547547","current":"Timeout","current_counter":99,"occurrences":{"Timeout":1293,"Pass":971,"DartkCrash":1},"first_seen":{"Timeout":"2020-03-08T23:24:14.639976","Pass":"2020-03-08T23:24:14.639976","DartkCrash":"2020-06-16T17:05:52.547547"},"last_seen":{"Timeout":"2020-08-07T23:22:24.226946","Pass":"2020-07-31T17:53:07.150471","DartkCrash":"2020-06-16T17:05:52.547547"},"matches":{"Timeout":false,"Pass":true,"DartkCrash":false},"build_ids":{"Timeout":"8872558816791850768","Pass":"8873213427860690816","DartkCrash":"8877293833016252192"},"commits":{"Timeout":"594434d5a0b530cfac750ca681ad059fd90d76fc","Pass":"1d1a775620e4715f590de8c1b93ddc0ca380893e","DartkCrash":"29e93bcdbdf0c3081999aa247bd3251bf58edbcd"}}

[jensjoha]

I found it here: https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket.appspot.com/8872368935561494880/+/steps/test_results/0/logs/ignored_flaky_test_failure_logs/0

[jensjoha]

On the dash-board you link to I can find:

• incremental_dart2js_test on Mac from March that timed out despite finishing (what?)
• incremental_compiler_leak_test on Linux, Windows and Mac from July, June and March. All seemingly being slow --- being nearly complete and having spend the allowed 4 minutes and being shut down by the test system.

How is it that one marks tests as slow these days?

[whesse]

I would put a line in https://github.com/dart-lang/sdk/blob/master/pkg/pkg.status
I'm not sure about what configuration I would put it with - probably where the other similar slow tests are in that file. It is no big disadvantage to mark it slow on more configurations than you need to, since it just allows it to take more time.