pkg/dds/test/dap/integration/dart_test_test flaky -> RuntimeError on unittest-asserts-release-mac-arm64

[bkonyi]

The tests

pkg/dds/test/dap/integration/dart_test_test RuntimeError (expected Pass)

are failing on configurations

unittest-asserts-release-mac-arm64

Logs.

[DanTup]

It seems like there are errors in the output when the debug adapter tries to run dart test (or dart run test:test):

STDERR output collected before app terminated:
Compilation did not produce any result. Expected file at `/Volumes/Work/s/w/itc7ufbxpn/dart-sdk-dap-testZbdZAx/appSGOmR5/.dart_tool/pub/bin/test/tmp88orE6/test.dart-3.5.0-edge.snapshot.incremental.temp`
Cannot rename file to '/Volumes/Work/s/w/itc7ufbxpn/dart-sdk-dap-testZbdZAx/appSGOmR5/.dart_tool/pub/bin/test/test.dart-3.5.0-edge.snapshot', path = '/Volumes/Work/s/w/itc7ufbxpn/dart-sdk-dap-testZbdZAx/appSGOmR5/.dart_tool/pub/bin/test/tmp88orE6/test.dart-3.5.0-edge.snapshot.incremental.temp' (OS Error: No such file or directory, errno = 2)
PathNotFoundException: Cannot rename file to '/Volumes/Work/s/w/itc7ufbxpn/dart-sdk-dap-testZbdZAx/appSGOmR5/.dart_tool/pub/bin/test/test.dart-3.5.0-edge.snapshot', path = '/Volumes/Work/s/w/itc7ufbxpn/dart-sdk-dap-testZbdZAx/appSGOmR5/.dart_tool/pub/bin/test/tmp88orE6/test.dart-3.5.0-edge.snapshot.incremental.temp' (OS Error: No such file or directory, errno = 2)

@bkonyi I'm not familiar with the file snapshot.incremental.temp, do you know who is creating and trying to rename it here? (the printed error doesn't seem to have any stack trace)

[bkonyi]

Ah, that's an artifact generated by pub when compiling the test runner. Maybe @sigurdm will have an idea as to what's going on?

[sigurdm]

Looks like compiling the test runner produced no output file.

Message is output from here:
https://github.com/dart-lang/pub/blob/be6868ba132c782d9835b1638a634ecb73428579/lib/src/dart.dart#L165

As to why that happens I have no clue.

[DanTup]

@bkonyi is there a way to see how often this is failing? (I'm curious if it was a one-off or happening a lot, and whether it's only on Mac ARM or not?)

[bkonyi]

@bkonyi is there a way to see how often this is failing? (I'm curious if it was a one-off or happening a lot, and whether it's only on Mac ARM or not?)

This particular instance appears to be failing on an ARM Mac, but we've seen a lot of failures on other configurations as well.

[DanTup]

I just reproduced this:

But I also saw this:

So I tried to run it:

Then clicked here:

And now I am unable to reproduce the failures.

@bkonyi @sigurdm could this be an infra thing? Could the newly-built dartaotruntime be being blocked by macOS?

[bkonyi]

I wonder if this is related to #55455? cc @sstrickl

[bkonyi]

Also FYI @athomas. Do we sign the bleeding edge dartaotruntime?

[HeySreelal]

Hi all, I ran into a similar issue (#55639) on Ubuntu server. Is there any fix or workaround for this yet? Thanks :)

[bkonyi]

@sigurdm is this possibly related to dart-lang/pub#4161 (comment)?

[sigurdm]

is this possibly related to dart-lang/pub#4161 (comment)?

Could be.... Are we running things in parallel here?

[athomas]

test.py runs tests in parallel, yes. So if multiple tests use pub, they might be running pub in parallel.

[DanTup]

Removing my assignment as I don't think this is something in DAP that I can fix (from the comments above, it either sounds like a concurrency issue or an issue with dartaotruntime).

Let me know if you think there is something I can do to fix this though.

[DanTup]

Oh, I just noticed that dart-lang/pub#4161 was closed (as fixed by dart-lang/pub#4170), so can it be the cause here?

[sigurdm]

Not that bug exactly. But I'm still not 100% confident that we have weeded out all problems with running dart run in parallel. Seems like a treasure trove of bugs.

[DanTup]

Ah, gotcha. @athomas is there a way to make these DAP tests run sequentially to see if that affects the results here?

[athomas]

The test runner just runs the main function, so it'd be possible to put them all in one file and ensure that the tests run one after the other. But if that "test of tests" gets too big you'll hit timeout issues. Maybe there are other ways to ensure the tests can run in parallel instead?

[DanTup]

To be clear - I don't think making the tests run sequentially is a real fix (because if the underlying issue is that running dart test concurrently has bugs, users and IDEs will hit those bugs even if these tests do not), but it would be useful to temporarily do that to see if that's really the issue here.

But now you've said that, I'm wondering if this can be caused by concurrency at all - only the tests in dart_test_test.dart are running dart test as far as I know, so it sounds like those are already running sequentially (just in parallel with other test files)?

(I am also still curious about #55453 (comment) above - are we certain that's not related - it was odd that I got exactly the same message running exactly the same test when I used a bleeding-edge build on a Mac, and this seemed to have been a recent development that I needed to tell macOS to allow that binary).

[athomas]

But now you've said that, I'm wondering if this can be caused by concurrency at all - only the tests in dart_test_test.dart are running dart test as far as I know, so it sounds like those are already running sequentially (just in parallel with other test files)?

Yes, other test files could run concurrently.

[athomas]

(I am also still curious about #55453 (comment) above - are we certain that's not related - it was odd that I got exactly the same message running exactly the same test when I used a bleeding-edge build on a Mac, and this seemed to have been a recent development that I needed to tell macOS to allow that binary).

The strange thing about the macOS stuff is that it's claiming chrome downloaded the binary. I'm wondering what dart you're using? Maybe try which dart. For SDK development, it should normally be a version built locally, not something downloaded via chrome. You may get different results when using tools/test.py.

[DanTup]

The strange thing about the macOS stuff is that it's claiming chrome downloaded the binary

Yeah, I downloaded the SDK from https://gsdview.appspot.com/dart-archive/channels/main/raw/latest/sdk/ (I tend to just grab these builds to work on the analysis server to avoid having to build the SDK myself frequently).

I don't know much about macOS security so I wasn't sure if a locally built version might trigger the same warning.. It's possibly completely unrelated, it just seemed odd I triggered the exact same error on the same test.

One way to verify if it's at all related could be to add a test that just runs dartaotruntime --version and ensures that works? Maybe there are already tests for this though Could be unrelated and a coincidence though - I'm just also not sure this is related to things running in parallel because I think only this one test suite runs dart test debug adapter tests.

[athomas]

The strange thing about the macOS stuff is that it's claiming chrome downloaded the binary

Yeah, I downloaded the SDK from https://gsdview.appspot.com/dart-archive/channels/main/raw/latest/sdk/ (I tend to just grab these builds to work on the analysis server to avoid having to build the SDK myself frequently).

I don't know much about macOS security so I wasn't sure if a locally built version might trigger the same warning.. It's possibly completely unrelated, it just seemed odd I triggered the exact same error on the same test.

One way to verify if it's at all related could be to add a test that just runs dartaotruntime --version and ensures that works? Maybe there are already tests for this though Could be unrelated and a coincidence though - I'm just also not sure this is related to things running in parallel because I think only this one test suite runs dart test debug adapter tests.

If you download an unsigned build (raw) from the internet (GCS) then the macOS warning is the expected behavior.

You could download signed dev releases instead, they are probably current enough for your purposes (sorry, no latest, you'd need to check the dev branch for the HEAD hash):
https://gsdview.appspot.com/dart-archive/channels/dev/signed/hash/

[DanTup]

It's no problem for me approving the unsigned version, I only brought it up because it resulted in the same error so I wondered if the issue here might be something similar (eg. macOS blocking it for some reason). If these security restrictions only apply to downloaded binaries, then sounds like it probably was just a coincidence.

[alexmarkov]

Happened once again:

00:04 �[32m+0�[0m�[31m -1�[0m: dart test can run without debugging �[1m�[31m[E]�[0m�[0m

  Expected: [
              '✓ group 1 passing test',
              'Expected: <2>',
              '  Actual: <1>',
              <(a string starting with 'package:matcher' and a string ending with 'expect')>,
              <a string ending with 'main.<fn>.<fn>'>,
              '✖ group 1 failing test',
              '! group 1 skipped test',
              '',
              'Exited (1).'
            ]
    Actual: [
              'Failed to load "/Volumes/Work/s/w/itu9b_l1b_/dart-sdk-dap-testlAtoby/apphghp6R/test_file.dart":',
              'package:test_core/src/runner/vm/platform.dart 242:7   VMPlatform._compileToKernel',
              '===== asynchronous gap ===========================',
              'package:test_core/src/runner/vm/platform.dart 220:13  VMPlatform._spawnIsolate',
              '===== asynchronous gap ===========================',
              'package:test_core/src/runner/vm/platform.dart 75:19   VMPlatform.load',
              '===== asynchronous gap ===========================',
              'package:test_core/src/runner/loader.dart 219:27       Loader.loadFile.<fn>',
              '===== asynchronous gap ===========================',
              'package:test_core/src/runner/load_suite.dart 98:19    new LoadSuite.<fn>.<fn>',
              'SocketException: Write failed (OS Error: Broken pipe, errno = 32), port = 0',
              'dart:io-patch/socket_patch.dart 1246:34                                _NativeSocket.write',
              'dart:io-patch/socket_patch.dart 2004:15                                _RawSocket.write',
              'dart:io-patch/socket_patch.dart 2481:18                                _Socket._write',
              'dart:io-patch/socket_patch.dart 2216:28                                _SocketStreamConsumer.write',
              'dart:io-patch/socket_patch.dart 2168:11                                _SocketStreamConsumer.addStream.<fn>',
              'dart:async/zone.dart 1407:47                                           _rootRunUnary',
              'dart:async/zone.dart 1308:19                                           _CustomZone.runUnary',
              'dart:async/zone.dart 1217:7                                            _CustomZone.runUnaryGuarded',
              'dart:async/stream_impl.dart 365:11                                     _BufferingStreamSubscription._sendData',
              'dart:async/stream_impl.dart 297:7                                      _BufferingStreamSubscription._add',
              'dart:async/stream_controller.dart 784:19                               _SyncStreamControllerDispatch._sendData',
              'dart:async/stream_controller.dart 658:7                                _StreamController._add',
              'dart:async/stream_controller.dart 606:5                                _StreamController.add',
              ...
            ]
     Which: at location [0] is 'Failed to load "/Volumes/Work/s/w/itu9b_l1b_/dart-sdk-dap-testlAtoby/apphghp6R/test_file.dart":' instead of '✓ group 1 passing test'
  

  package:matcher/src/expect/expect.dart 149:31          fail
  package:matcher/src/expect/expect.dart 144:3           _expect
  package:matcher/src/expect/expect.dart 56:3            expect
  pkg/dds/test/dap/integration/test_support.dart 54:3    expectLines
  pkg/dds/test/dap/integration/dart_test_test.dart 71:7  main.<fn>.<fn>
  ===== asynchronous gap ===========================
  package:test_api/src/backend/declarer.dart 215:9       Declarer.test.<fn>.<fn>
  ===== asynchronous gap ===========================
  package:test_api/src/backend/declarer.dart 213:7       Declarer.test.<fn>
  ===== asynchronous gap ===========================
  package:test_api/src/backend/invoker.dart 258:9        Invoker._waitForOutstandingCallbacks.<fn>

Log: https://dart-ci.appspot.com/log/pkg-mac-release-arm64/unittest-asserts-release-mac-arm64/12957/pkg/dds/test/dap/integration/dart_test_test

[DanTup]

That error is thrown here:

https://github.com/dart-lang/test/blob/2464ad5c5945c98edd33fb69b3616a14771f1c8d/pkgs/test_core/lib/src/runner/vm/platform.dart#L242

And I guess the compilerOutput was:

SocketException: Write failed (OS Error: Broken pipe, errno = 32), port =

Given the code, I'm not sure if this is DAP-related, or something is just going wrong with the execution of dart test test_file.dart.

Do you have any idea how often it fails and whether it's only the ARM bot? (I presume there are other dart test tests that are not flaking with this kind of failure?)