service_2/breakpoint_in_package_parts_class_file_uri_test went flaky -> RuntimeError

[alexmarkov]

The tests

service_2/breakpoint_in_package_parts_class_file_uri_test/dds RuntimeError (expected Pass)
service_2/breakpoint_in_package_parts_class_file_uri_test/service RuntimeError (expected Pass)

are failing on configurations

app_jitk-linux-release-x64

The tests were flaky until recently.

From log:

>testee>out> Instance of 'Foo10'

Running out.jitsnapshot [4/4]

00:00 �[32m+0�[0m�[31m -1�[0m: out.jitsnapshot (DDS) �[1m�[31m[E]�[0m�[0m

  Expected: <true>
    Actual: <false>
  Expects at least 2 breaks, got 0.
  

  package:test_api/src/frontend/expect.dart 155:31                       fail
  package:test_api/src/frontend/expect.dart 150:3                        _expect
  package:test_api/src/frontend/expect.dart 59:3                         expect
  runtime/observatory_2/tests/service_2/service_test_common.dart 603:5   checkRecordedStops.<fn>
  runtime/observatory_2/tests/service_2/service_test_common.dart 552:10  checkRecordedStops.<fn>
  runtime/observatory_2/tests/service_2/test_helper.dart 455:27          _ServiceTesterRunner.run.runTest.<fn>.<fn>
  ===== asynchronous gap ===========================
  dart:async/zone.dart 1294:19                                           _CustomZone.registerBinaryCallback
  dart:async-patch/async_patch.dart 61:8                                 _asyncErrorWrapperHelper
  package:test_api/src/backend/invoker.dart                              Invoker.waitForOutstandingCallbacks.<fn>
  dart:async/zone.dart 1354:13                                           _rootRun
  dart:async/zone.dart 1258:19                                           _CustomZone.run
  dart:async/zone.dart 1789:10                                           _runZoned
  dart:async/zone.dart 1711:10                                           runZoned
  package:test_api/src/backend/invoker.dart 228:5                        Invoker.waitForOutstandingCallbacks
  package:test_api/src/backend/invoker.dart 383:17                       Invoker._onRun.<fn>.<fn>.<fn>
  ===== asynchronous gap ===========================
  dart:async/zone.dart 1286:19                                           _CustomZone.registerUnaryCallback
  dart:async-patch/async_patch.dart 45:22                                _asyncThenWrapperHelper
  package:test_api/src/backend/invoker.dart                              Invoker._onRun.<fn>.<fn>.<fn>
  dart:async/zone.dart 1354:13                                           _rootRun
  dart:async/zone.dart 1258:19                                           _CustomZone.run
  dart:async/zone.dart 1789:10                                           _runZoned
  dart:async/zone.dart 1711:10                                           runZoned
  package:test_api/src/backend/invoker.dart 370:9                        Invoker._onRun.<fn>.<fn>
  package:test_api/src/backend/invoker.dart 415:15                       Invoker._guardIfGuarded
  package:test_api/src/backend/invoker.dart 369:7                        Invoker._onRun.<fn>
  package:stack_trace/src/chain.dart 94:24                               Chain.capture.<fn>
  dart:async/zone.dart 1354:13                                           _rootRun
  dart:async/zone.dart 1258:19                                           _CustomZone.run
  dart:async/zone.dart 1789:10                                           _runZoned
  dart:async/zone.dart 1711:10                                           runZoned
  package:stack_trace/src/chain.dart 92:12                               Chain.capture
  package:test_api/src/backend/invoker.dart 368:11                       Invoker._onRun
  package:test_api/src/backend/live_test_controller.dart 153:11          LiveTestController.run
  dart:async/future.dart 198:37                                          new Future.microtask.<fn>
  dart:async/zone.dart 1346:47                                           _rootRun
  dart:async/zone.dart 1258:19                                           _CustomZone.run
  dart:async/zone.dart 1162:7                                            _CustomZone.runGuarded
  dart:async/zone.dart 1202:23                                           _CustomZone.bindCallbackGuarded.<fn>
  dart:async/zone.dart 1354:13                                           _rootRun
  dart:async/zone.dart 1258:19                                           _CustomZone.run
  dart:async/zone.dart 1162:7                                            _CustomZone.runGuarded
  dart:async/zone.dart 1202:23                                           _CustomZone.bindCallbackGuarded.<fn>
  dart:async/schedule_microtask.dart 40:21                               _microtaskLoop
  dart:async/schedule_microtask.dart 49:5                                _startMicrotaskLoop
  dart:isolate-patch/isolate_patch.dart 120:13                           _runPendingImmediateCallback
  dart:isolate-patch/timer_impl.dart 402:11                              _Timer._runTimers
  dart:isolate-patch/timer_impl.dart 426:5                               _Timer._handleMessage
  dart:isolate-patch/isolate_patch.dart 184:12                           _RawReceivePortImpl._handleMessage

Logs:
https://dart-ci.appspot.com/log/app-kernel-linux-release-x64/app_jitk-linux-release-x64/10713/service_2/breakpoint_in_package_parts_class_file_uri_test/dds
https://dart-ci.appspot.com/log/app-kernel-linux-release-x64/app_jitk-linux-release-x64/10713/service_2/breakpoint_in_package_parts_class_file_uri_test/service

@bkonyi

[alexmarkov]

Turned from flaky to RuntimeError again.

Recent logs:

--- Command "app_jit" (took 37.000326s):
DART_CONFIGURATION=ReleaseX64 out/ReleaseX64/dart --snapshot=/b/s/w/ir/cache/builder/sdk/out/ReleaseX64/generated_compilations/app_jitk-linux-release-x64/runtime_observatory_2_tests_service_2_breakpoint_in_package_parts_class_file_uri_test/out.jitsnapshot --snapshot-kind=app-jit --ignore-unrecognized-flags --packages=/b/s/w/ir/cache/builder/sdk/.packages /b/s/w/ir/cache/builder/sdk/runtime/observatory_2/tests/service_2/breakpoint_in_package_parts_class_file_uri_test.dart

exit code:
0

stdout:
Info: Compiling without sound null safety
00:00 �[32m+0�[0m: breakpoint_in_package_parts_class_file_uri_test.dart (VM Service)�[0m

** Launching SERVICE_TEST_TESTEE=true out/ReleaseX64/dart --disable-dart-dev --pause-isolates-on-start --pause-isolates-on-exit --write-service-info=file:///b/s/w/ir/x/t/dart_serviceEJXIAG/service_info.json --profiler --snapshot=/b/s/w/ir/cache/builder/sdk/out/ReleaseX64/generated_compilations/app_jitk-linux-release-x64/runtime_observatory_2_tests_service_2_breakpoint_in_package_parts_class_file_uri_test/out.jitsnapshot --snapshot-kind=app-jit --ignore-unrecognized-flags --packages=/b/s/w/ir/cache/builder/sdk/.packages --enable-vm-service:0 /b/s/w/ir/cache/builder/sdk/runtime/observatory_2/tests/service_2/breakpoint_in_package_parts_class_file_uri_test.dart

>testee>out> Observatory listening on http://127.0.0.1:37371/OSgKRL83Pow=/

** Signaled to run test queries on http://127.0.0.1:37371/OSgKRL83Pow= (VM Service)

Loading VM...

Done loading VM

Running breakpoint_in_package_parts_class_file_uri_test.dart [1/4]

Paused with PauseStart

Running breakpoint_in_package_parts_class_file_uri_test.dart [2/4]

Setting breakpoint for line 87 in file:///b/s/w/ir/cache/builder/sdk/runtime/observatory_2/tests/service_2/observatory_test_package_2/the_part.dart

Breakpoint is Breakpoint 1 at the_part.dart:87[unresolved]

Running breakpoint_in_package_parts_class_file_uri_test.dart [3/4]

>testee>out> hello from foo!

>testee>out> Instance of 'Foo10'

Running breakpoint_in_package_parts_class_file_uri_test.dart [4/4]

All service tests completed successfully.

** Killing script

00:17 �[32m+1�[0m: All tests passed!�[0m

** Process exited

--- Command "vm" (took 01.000556s):
DART_CONFIGURATION=ReleaseX64 out/ReleaseX64/dart --ignore-unrecognized-flags --packages=/b/s/w/ir/cache/builder/sdk/.packages /b/s/w/ir/cache/builder/sdk/out/ReleaseX64/generated_compilations/app_jitk-linux-release-x64/runtime_observatory_2_tests_service_2_breakpoint_in_package_parts_class_file_uri_test/out.jitsnapshot

exit code:
255

stdout:
00:00 �[32m+0�[0m: out.jitsnapshot (VM Service)�[0m

** Launching SERVICE_TEST_TESTEE=true out/ReleaseX64/dart --disable-dart-dev --pause-isolates-on-start --pause-isolates-on-exit --write-service-info=file:///b/s/w/ir/x/t/dart_serviceQBJAJS/service_info.json --profiler --ignore-unrecognized-flags --packages=/b/s/w/ir/cache/builder/sdk/.packages --enable-vm-service:0 /b/s/w/ir/cache/builder/sdk/out/ReleaseX64/generated_compilations/app_jitk-linux-release-x64/runtime_observatory_2_tests_service_2_breakpoint_in_package_parts_class_file_uri_test/out.jitsnapshot

>testee>out> Observatory listening on http://127.0.0.1:37249/G60R1B4_rOk=/

** Signaled to run test queries on http://127.0.0.1:37249/G60R1B4_rOk= (VM Service)

Loading VM...

Done loading VM

Running out.jitsnapshot [1/4]

Paused with PauseStart

Running out.jitsnapshot [2/4]

Setting breakpoint for line 87 in file:///b/s/w/ir/cache/builder/sdk/out/ReleaseX64/generated_compilations/app_jitk-linux-release-x64/runtime_observatory_2_tests_service_2_breakpoint_in_package_parts_class_file_uri_test/observatory_test_package_2/the_part.dart

Breakpoint is Breakpoint 1 at file:///b/s/w/ir/cache/builder/sdk/out/ReleaseX64/generated_compilations/app_jitk-linux-release-x64/runtime_observatory_2_tests_service_2_breakpoint_in_package_parts_class_file_uri_test/observatory_test_package_2/the_part.dart:87[unresolved]

Running out.jitsnapshot [3/4]

>testee>out> hello from foo!

>testee>out> Instance of 'Foo10'

Running out.jitsnapshot [4/4]

00:00 �[32m+0�[0m�[31m -1�[0m: out.jitsnapshot (VM Service) �[1m�[31m[E]�[0m�[0m

  Expected: <true>
    Actual: <false>
  Expects at least 2 breaks, got 0.
  

  package:test_api/src/expect/expect.dart 137:31                         fail
  package:test_api/src/expect/expect.dart 132:3                          _expect
  package:test_api/src/expect/expect.dart 46:3                           expect
  runtime/observatory_2/tests/service_2/service_test_common.dart 605:5   checkRecordedStops.<fn>
  runtime/observatory_2/tests/service_2/service_test_common.dart 554:10  checkRecordedStops.<fn>
  runtime/observatory_2/tests/service_2/test_helper.dart 457:27          _ServiceTesterRunner.run.runTest.<fn>.<fn>
  ===== asynchronous gap ===========================
  runtime/observatory_2/tests/service_2/test_helper.dart 356:9           _ServiceTesterRunner.run.ignoreLateException
  ===== asynchronous gap ===========================
  package:test_api/src/backend/declarer.dart 199:9                       Declarer.test.<fn>.<fn>
  ===== asynchronous gap ===========================
  package:test_api/src/backend/declarer.dart 197:7                       Declarer.test.<fn>
  ===== asynchronous gap ===========================
  package:test_api/src/backend/invoker.dart 257:7                        Invoker._waitForOutstandingCallbacks.<fn>
  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
** Killing script

00:00 �[32m+0�[0m�[31m -1�[0m: �[31mSome tests failed.�[0m

stderr:
Unhandled exception:
Dummy exception to set exit code.

--- Re-run this test:
python3 tools/test.py -n app_jitk-linux-release-x64 service_2/breakpoint_in_package_parts_class_file_uri_test/service

https://dart-ci.appspot.com/log/app-kernel-linux-release-x64/app_jitk-linux-release-x64/11924/service_2/breakpoint_in_package_parts_class_file_uri_test/service
https://dart-ci.appspot.com/log/app-kernel-linux-release-x64/app_jitk-linux-release-x64/11924/service_2/breakpoint_in_package_parts_class_file_uri_test/dds

[bkonyi]

Happened again, same error. Marking test as RuntimeError again.

[alexmarkov]

This test also turned from flaky to RuntimeError on dartk-linux-release-simarm64 configuration.

Log:

--- Command "vm_compile_to_kernel" (took 08.000414s):
DART_CONFIGURATION=ReleaseSIMARM64 /b/s/w/ir/pkg/vm/tool/gen_kernel --no-aot --platform=out/ReleaseSIMARM64/vm_platform_strong.dill -o /b/s/w/ir/out/ReleaseSIMARM64/generated_compilations/dartk-linux-release-simarm64/runtime_observatory_2_tests_service_2_breakpoint_in_package_parts_class_file_uri_test/out.dill /b/s/w/ir/runtime/observatory_2/tests/service_2/breakpoint_in_package_parts_class_file_uri_test.dart -Dtest_runner.configuration=dartk-linux-release-simarm64 --packages=/b/s/w/ir/.dart_tool/package_config.json -Ddart.vm.product=false

exit code:
0

--- Command "vm" (took 09.000608s):
DART_CONFIGURATION=ReleaseSIMARM64 out/ReleaseSIMARM64/dart -Dtest_runner.configuration=dartk-linux-release-simarm64 --ignore-unrecognized-flags --packages=/b/s/w/ir/.dart_tool/package_config.json /b/s/w/ir/out/ReleaseSIMARM64/generated_compilations/dartk-linux-release-simarm64/runtime_observatory_2_tests_service_2_breakpoint_in_package_parts_class_file_uri_test/out.dill

exit code:
255

stdout:
00:00 �[32m+0�[0m: out.dill (VM Service)�[0m

** Launching SERVICE_TEST_TESTEE=true out/ReleaseSIMARM64/dart --disable-dart-dev --pause-isolates-on-start --pause-isolates-on-exit --write-service-info=file:///b/s/w/itedz43yuy/dart_serviceQVZXTV/service_info.json --profiler -Dtest_runner.configuration=dartk-linux-release-simarm64 --ignore-unrecognized-flags --packages=/b/s/w/ir/.dart_tool/package_config.json --enable-vm-service:0 /b/s/w/ir/out/ReleaseSIMARM64/generated_compilations/dartk-linux-release-simarm64/runtime_observatory_2_tests_service_2_breakpoint_in_package_parts_class_file_uri_test/out.dill

>testee>out> The Dart VM service is listening on http://127.0.0.1:45401/3PKJOEn7ZTg=/

** Signaled to run test queries on http://127.0.0.1:45401/3PKJOEn7ZTg= (VM Service)

Loading VM...

Done loading VM

Running out.dill [1/4]

Paused with PauseStart

Running out.dill [2/4]

Setting breakpoint for line 87 in file:///b/s/w/ir/out/ReleaseSIMARM64/generated_compilations/dartk-linux-release-simarm64/runtime_observatory_2_tests_service_2_breakpoint_in_package_parts_class_file_uri_test/observatory_test_package_2/the_part.dart

Breakpoint is Breakpoint 1 at file:///b/s/w/ir/out/ReleaseSIMARM64/generated_compilations/dartk-linux-release-simarm64/runtime_observatory_2_tests_service_2_breakpoint_in_package_parts_class_file_uri_test/observatory_test_package_2/the_part.dart:87[unresolved]

Running out.dill [3/4]

>testee>out> hello from foo!

>testee>out> Instance of 'Foo10'

Running out.dill [4/4]

00:07 �[32m+0�[0m�[31m -1�[0m: out.dill (VM Service) �[1m�[31m[E]�[0m�[0m

  Expected: <true>
    Actual: <false>
  Expects at least 2 breaks, got 0.
  

  package:test_api/src/expect/expect.dart 134:31                        fail
  package:test_api/src/expect/expect.dart 129:3                         _expect
  package:test_api/src/expect/expect.dart 46:3                          expect
  runtime/observatory_2/tests/service_2/service_test_common.dart 598:5  checkRecordedStops.<fn>
  runtime/observatory_2/tests/service_2/test_helper.dart 495:27         _ServiceTesterRunner.run.runTest.<fn>.<fn>
  ===== asynchronous gap ===========================
  runtime/observatory_2/tests/service_2/test_helper.dart 374:9          _ServiceTesterRunner.run.ignoreLateException
  ===== asynchronous gap ===========================
  package:test_api/src/backend/declarer.dart 215:9                      Declarer.test.<fn>.<fn>
  ===== asynchronous gap ===========================
  package:test_api/src/backend/declarer.dart 213:7                      Declarer.test.<fn>
  ===== asynchronous gap ===========================
  package:test_api/src/backend/invoker.dart 257:7                       Invoker._waitForOutstandingCallbacks.<fn>
  

** Killing script

00:07 �[32m+0�[0m�[31m -1�[0m: �[31mSome tests failed.�[0m



Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.

stderr:
Unhandled exception:
Dummy exception to set exit code.

--- Re-run this test:
python3 tools/test.py -n dartk-linux-release-simarm64 service_2/breakpoint_in_package_parts_class_file_uri_test/service

https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket/8806648128650934225/+/u/test_results/stdout
https://dart-ci.appspot.com/log/vm-kernel-linux-release-simarm64/dartk-linux-release-simarm64/14417/service_2/breakpoint_in_package_parts_class_file_uri_test/dds

[derekxu16]

The test also went from flaky to RuntimeError on app_jitk-linux-debug-x64

https://ci.chromium.org/ui/p/dart/builders/ci.sandbox/app-kernel-linux-debug-x64/11959/overview

[derekxu16]

This happened again on vm-kernel-nnbd-linux-release-simarm and vm-kernel-nnbd-linux-release-simarm64

https://ci.chromium.org/ui/p/dart/builders/ci.sandbox/vm-kernel-nnbd-linux-release-simarm/766/overview
https://ci.chromium.org/ui/p/dart/builders/ci.sandbox/vm-kernel-nnbd-linux-release-simarm64/766/overview

[dcharkes]

More meta-flaking.

https://dart-ci.firebaseapp.com/#showLatestFailures=false&test=service/breakpoint_in_package_parts_class_file_uri_test/dds
and
https://dart-ci.firebaseapp.com/#showLatestFailures=false&test=service/breakpoint_in_package_parts_class_file_uri_test/service

Both logs mention

  Expects at least 2 breaks, got 0.

[dcharkes]

Reapproved:

https://dart-ci.appspot.com/log/vm-appjit-linux-release-x64/vm-appjit-linux-release-x64/1035/service/breakpoint_in_package_parts_class_file_uri_test/dds

[a-siva]

I noticed the test is written to set breakpoints as follows

String baseDirectory = path.dirname(Platform.script.path) + '/';
Uri baseUri = Platform.script.replace(path: baseDirectory);
Uri breakpointFile = baseUri.resolve('observatory_test_package/the_part.dart');

When run from an app-jit snapshot this results in an attempt to set a breakpoint with the script url as

file:///usr/local/google/home/asiva/workspace/dart-tot/sdk/out/ReleaseX64/generated_compilations/vm-appjit-linux-release-x64/runtime_observatory_tests_service_breakpoint_in_package_parts_class_file_uri_test.dart_1/observatory_test_package/the_part.dart

No such script exists in the app and so we end up setting a latent breakpoint which is never resolved.
Going to mark this test as an invalid test in the app-jit configuration.

[a-siva]

Status file adjustment is here https://dart-review.googlesource.com/c/sdk/+/312888