Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

break_on_function_many_child_isolates_test flakily fails on app_jitk-linux-debug-x64 with enable-isolate-groups #46909

Closed
aam opened this issue Aug 13, 2021 · 4 comments
Closed

break_on_function_many_child_isolates_test flakily fails on app_jitk-linux-debug-x64 with enable-isolate-groups #46909

aam opened this issue Aug 13, 2021 · 4 comments
Assignees
@rmacnak-google
Labels
area-vm Use area-vm for VM related issues, including code coverage, and the AOT and JIT backends. gardening

Comments

@aam
Copy link
Contributor

aam commented Aug 13, 2021 •
edited
Loading

from logs

/================================================================================================================\
| service_2/break_on_function_many_child_isolates_test/dds failed again (Timeout -> RuntimeError, expected Pass) |
\================================================================================================================/

--- Command "app_jit" (took 01:38.000685s):
DART_CONFIGURATION=DebugX64 out/DebugX64/dart --snapshot=/b/s/w/ir/out/DebugX64/generated_compilations/app_jitk-linux-debug-x64/runtime_observatory_2_tests_service_2_break_on_function_many_child_isolates_test.dart_1/out.jitsnapshot --snapshot-kind=app-jit --verbose_debug --enable-isolate-groups -DUSE_DDS=true --ignore-unrecognized-flags --packages=/b/s/w/ir/.packages /b/s/w/ir/runtime/observatory_2/tests/service_2/break_on_function_many_child_isolates_test.dart

exit code:
255

stdout:
Info: Compiling without sound null safety
00:00 �[32m+0�[0m: break_on_function_many_child_isolates_test.dart (DDS)�[0m

** Launching SERVICE_TEST_TESTEE=true out/DebugX64/dart --disable-dart-dev --pause-isolates-on-start --write-service-info=file:///b/s/w/it67emj6c8/dart_serviceVCLDMT/service_info.json --profiler --snapshot=/b/s/w/ir/out/DebugX64/generated_compilations/app_jitk-linux-debug-x64/runtime_observatory_2_tests_service_2_break_on_function_many_child_isolates_test.dart_1/out.jitsnapshot --snapshot-kind=app-jit --verbose_debug --enable-isolate-groups -DUSE_DDS=true --ignore-unrecognized-flags --packages=/b/s/w/ir/.packages --enable-vm-service:0 /b/s/w/ir/runtime/observatory_2/tests/service_2/break_on_function_many_child_isolates_test.dart

>testee>out> Observatory listening on http://127.0.0.1:42565/c0PuV3iSIGI=/

** Signaled to run test queries on http://127.0.0.1:46381/292v5kHoRrw=/ (DDS)

Loading VM...

Done loading VM

Running break_on_function_many_child_isolates_test.dart [1/10]

Paused with PauseStart

Running break_on_function_many_child_isolates_test.dart [2/10]

Running break_on_function_many_child_isolates_test.dart [3/10]

>testee>err> HandleSteppingRequest- kStepOut 7f23eaffe298

>testee>err> >>> single step break at file:///b/s/w/ir/runtime/observatory_2/tests/service_2/break_on_function_many_child_isolates_test.dart:36:14 (func :async_op token 1068 address 0x7f23f61a6b85 offset 0x375)

Paused with PauseBreakpoint

Running break_on_function_many_child_isolates_test.dart [4/10]

Checking we are at line 36

Program is stopped at line: 36

Running break_on_function_many_child_isolates_test.dart [5/10]

Running break_on_function_many_child_isolates_test.dart [6/10]

Running break_on_function_many_child_isolates_test.dart [7/10]

>testee>err> ../../runtime/vm/raw_object.h: 332: error: expected: !this->IsRemembered()

>testee>err> version=2.15.0-edge.e8ddc0219f1e8f1ad784143fec693890e2b81954 (be) (Fri Aug 13 13:27:41 2021 +0000) on "linux_x64"

>testee>err> pid=26471, thread=26651, isolate_group=main(0x56098b672800), isolate=foo2(0x56098ba01000)

>testee>err> isolate_instructions=56098978b080, vm_instructions=56098978b080

>testee>err> pc 0x0000560989b82a6c fp 0x00007f23f1977540 dart::Profiler::DumpStackTrace(void*)+0x7c

>testee>err> pc 0x000056098978b234 fp 0x00007f23f1977620 dart::Assert::Fail(char const*, ...)+0x84

>testee>err> pc 0x000056098991c85a fp 0x00007f23f1977660 out/DebugX64/dart+0x1fde85a

>testee>err> pc 0x0000560989ad52e3 fp 0x00007f23f19776d0 dart::Library::LoadedScripts() const+0x573

>testee>err> pc 0x0000560989ad4c72 fp 0x00007f23f1977730 dart::Script::FindLibrary() const+0xe2

>testee>err> pc 0x0000560989b6cb6b fp 0x00007f23f1977790 dart::Script::PrintJSONImpl(dart::JSONStream*, bool) const+0x9b

>testee>err> pc 0x0000560989b6db2d fp 0x00007f23f1977830 dart::Library::PrintJSONImpl(dart::JSONStream*, bool) const+0xb4d

>testee>err> pc 0x0000560989c094b2 fp 0x00007f23f1977880 out/DebugX64/dart+0x22cb4b2

>testee>err> pc 0x0000560989c02278 fp 0x00007f23f1977a00 dart::Service::InvokeMethod(dart::Isolate*, dart::Array const&, bool)+0x4f8

>testee>err> pc 0x0000560989c02d0e fp 0x00007f23f1977a30 dart::Service::HandleIsolateMessage(dart::Isolate*, dart::Array const&)+0x1e

>testee>err> pc 0x0000560989a1359c fp 0x00007f23f1977b20 dart::IsolateMessageHandler::HandleMessage(std::__2::unique_ptr<dart::Message, std::__2::default_delete<dart::Message> >)+0x35c

>testee>err> pc 0x0000560989a56b48 fp 0x00007f23f1977bb0 dart::MessageHandler::HandleMessages(dart::MonitorLocker*, bool, bool)+0x218

>testee>err> pc 0x0000560989a576fd fp 0x00007f23f1977c30 dart::MessageHandler::TaskCallback()+0x11d

>testee>err> pc 0x0000560989c3a3cb fp 0x00007f23f1977cd0 dart::ThreadPool::WorkerLoop(dart::ThreadPool::Worker*)+0x1fb

>testee>err> pc 0x0000560989c3ad67 fp 0x00007f23f1977d20 dart::ThreadPool::Worker::Main(unsigned long)+0x147

>testee>err> pc 0x0000560989b79f84 fp 0x00007f23f1977e70 out/DebugX64/dart+0x223bf84

>testee>err> -- End of DumpStackTrace

00:50 �[32m+0�[0m�[31m -1�[0m: break_on_function_many_child_isolates_test.dart (DDS) �[1m�[31m[E]�[0m�[0m

  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
  ServerRpcException(Service has disappeared)

  

Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.
** Killing script

00:50 �[32m+0�[0m�[31m -1�[0m: �[31mSome tests failed.�[0m

stderr:
Unhandled exception:
Dummy exception to set exit code.

--- Re-run this test:
python3 tools/test.py -n app_jitk-linux-debug-x64 service_2/break_on_function_many_child_isolates_test/dds
@aam aam added area-vm Use area-vm for VM related issues, including code coverage, and the AOT and JIT backends. gardening labels Aug 13, 2021
@aam
Copy link
Contributor Author

aam commented Aug 16, 2021

cc @rmacnak-google @mkustermann

@mkustermann
Copy link
Member

Seems like the C++ code is not correct

  DART_FORCE_INLINE
  void AddToRememberedSet(Thread* thread) {
    ASSERT(!this->IsRemembered());
    this->SetRememberedBit();
    thread->StoreBufferAddObject(ObjectPtr(this));
  }

 void CheckHeapPointerStore(ObjectPtr value, Thread* thread) {
    ...
      if (value->IsNewObject()) {
        AddToRememberedSet(thread);
    ...
  }
  template <typename type>
  void StorePointer(type const* addr, type value, Thread* thread) {
    *const_cast<type*>(addr) = value;
    if (value.IsHeapObject()) {
      CheckHeapPointerStore(value, thread);
    }
  }

It should do a EnsureIsInRememberedSet() instead with a compare-and-swap, only the thread that manages to set the bit should add it to the set.

dart-bot pushed a commit that referenced this issue Aug 16, 2021
@rmacnak-google
[vm, gc] Make the generational barriers in the runtime multi-thread s…
a3bcca9 
…afe.

TEST=ci
Bug: #46909
Change-Id: I9e26b88779ff9039ab7723d07ef9f8568e6595d8
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/210283
Commit-Queue: Ryan Macnak <rmacnak@google.com>
Reviewed-by: Martin Kustermann <kustermann@google.com>
@aam aam closed this as completed Aug 19, 2021
dart-bot pushed a commit that referenced this issue Aug 20, 2021
@mraleph
Revert "[vm, gc] Make the generational barriers in the runtime multi-…
a2e5dc8 
…thread safe."

This reverts commit a3bcca9.

Reason for revert: there are indications that this is causing flaky crashes (b/196919000)

Original change's description:
> [vm, gc] Make the generational barriers in the runtime multi-thread safe.
>
> TEST=ci
> Bug: #46909
> Change-Id: I9e26b88779ff9039ab7723d07ef9f8568e6595d8
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/210283
> Commit-Queue: Ryan Macnak <rmacnak@google.com>
> Reviewed-by: Martin Kustermann <kustermann@google.com>

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug: #46909
Change-Id: Ifdbf921dbcf439ba4dfa77a63a37b34c6bfe815c
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/210682
Reviewed-by: Martin Kustermann <kustermann@google.com>
Commit-Queue: Slava Egorov <vegorov@google.com>
@mraleph
Copy link
Member

mraleph commented Aug 24, 2021

Reopening the bug because the fix was reverted due to b/196919000

@mraleph mraleph reopened this Aug 24, 2021
dart-bot pushed a commit that referenced this issue Aug 24, 2021
@rmacnak-google
[vm, gc] Make the generational barriers in the runtime multi-thread s…
a81945e 
…afe.

TEST=ci
Bug: #46909
Change-Id: I24a93c0293e9b559d113bc504f731fb2df24f82a
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/211080
Reviewed-by: Siva Annamalai <asiva@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
dart-bot pushed a commit that referenced this issue Aug 25, 2021
@iinozemtsev
Revert "[vm, gc] Make the generational barriers in the runtime multi-…
7da6815 
…thread safe."

This reverts commit a81945e.

Reason for revert: b/196919000#comment34

Original change's description:
> [vm, gc] Make the generational barriers in the runtime multi-thread safe.
>
> TEST=ci
> Bug: #46909
> Change-Id: I24a93c0293e9b559d113bc504f731fb2df24f82a
> Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/211080
> Reviewed-by: Siva Annamalai <asiva@google.com>
> Commit-Queue: Ryan Macnak <rmacnak@google.com>

TBR=rmacnak@google.com,asiva@google.com

Change-Id: Idb925c2c639cb00952c7d4fecfb98582270a36ee
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: #46909
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/211242
Reviewed-by: Slava Egorov <vegorov@google.com>
Commit-Queue: Slava Egorov <vegorov@google.com>
@mkustermann
Copy link
Member

It seems like this has been reverted again, so I'll reopen the bug.

@mkustermann mkustermann reopened this Aug 30, 2021
dart-bot pushed a commit that referenced this issue Sep 9, 2021
@rmacnak-google
[vm, gc] Make the generational barriers in the runtime multi-thread s…
2f8e129 
…afe.

TEST=ci
Bug: #46909
Change-Id: I85b1cd2a7fe7aa835b2dc6b6232dee412219f3ce
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/212029
Reviewed-by: Siva Annamalai <asiva@google.com>
Commit-Queue: Ryan Macnak <rmacnak@google.com>
@liamappelbe liamappelbe mentioned this issue Sep 30, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rmacnak-google rmacnak-google

Labels
area-vm Use area-vm for VM related issues, including code coverage, and the AOT and JIT backends. gardening
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mraleph @aam @mkustermann @rmacnak-google