Dangling pointer observed in service heap snapshot test - possible compactor bug #34934

rmacnak-google · 2018-10-25T20:22:13Z

FAILED: dartk-vm release_x64 service/dominator_tree_vm_test
Expected: Pass
Actual: RuntimeError

--- Command "vm" (took 10.000369s):
DART_CONFIGURATION=ReleaseX64 out/ReleaseX64/dart --use_compactor --optimization-counter-threshold=5 --ignore-unrecognized-flags --packages=/b/s/w/ir/.packages /b/s/w/ir/runtime/observatory/tests/service/dominator_tree_vm_test.dart

exit code:
255

stdout:
** Launching SERVICE_TEST_TESTEE=true out/ReleaseX64/dart -DDART_SERVICE_USE_AUTH=false --profiler --use_compactor --optimization-counter-threshold=5 --ignore-unrecognized-flags --packages=/b/s/w/ir/.packages --enable-vm-service:0 /b/s/w/ir/runtime/observatory/tests/service/dominator_tree_vm_test.dart
>testee>out> Observatory listening on http://127.0.0.1:42819/
** Signaled to run test queries on http://127.0.0.1:42819/
>testee>out> 
Loading VM...
Done loading VM
Running dominator_tree_vm_test.dart [1/1]
** Killing script
Unexpected exception in service tests: Exception: Heap snapshot contains an edge but lacks its target node
#0      ObjectGraph._remapEdges (package:observatory/object_graph.dart:674:11)
#1      ObjectGraph.process.<anonymous closure>.<anonymous closure> (package:observatory/object_graph.dart:498:30)
#2      new Future.<anonymous closure> (dart:async/future.dart:176:37)
#3      _rootRun (dart:async/zone.dart:1120:38)
#4      _CustomZone.run (dart:async/zone.dart:1021:19)
#5      _CustomZone.runGuarded (dart:async/zone.dart:923:7)
#6      _CustomZone.bindCallbackGuarded.<anonymous closure> (dart:async/zone.dart:963:23)
#7      _rootRun (dart:async/zone.dart:1124:13)
#8      _CustomZone.run (dart:async/zone.dart:1021:19)
#9      _CustomZone.bindCallback.<anonymous closure> (dart:async/zone.dart:947:23)
#10     Timer._createTimer.<anonymous closure> (dart:async/runtime/libtimer_patch.dart:21:15)
#11     _Timer._runTimers (dart:isolate/runtime/libtimer_impl.dart:382:19)
#12     _Timer._handleMessage (dart:isolate/runtime/libtimer_impl.dart:416:5)
#13     _RawReceivePortImpl._handleMessage (dart:isolate/runtime/libisolate_patch.dart:171:12)

stderr:
Unhandled exception:
Exception: Heap snapshot contains an edge but lacks its target node
#0      ObjectGraph._remapEdges (package:observatory/object_graph.dart:674:11)
#1      ObjectGraph.process.<anonymous closure>.<anonymous closure> (package:observatory/object_graph.dart:498:30)
#2      new Future.<anonymous closure> (dart:async/future.dart:176:37)
#3      _rootRun (dart:async/zone.dart:1120:38)
#4      _CustomZone.run (dart:async/zone.dart:1021:19)
#5      _CustomZone.runGuarded (dart:async/zone.dart:923:7)
#6      _CustomZone.bindCallbackGuarded.<anonymous closure> (dart:async/zone.dart:963:23)
#7      _rootRun (dart:async/zone.dart:1124:13)
#8      _CustomZone.run (dart:async/zone.dart:1021:19)
#9      _CustomZone.bindCallback.<anonymous closure> (dart:async/zone.dart:947:23)
#10     Timer._createTimer.<anonymous closure> (dart:async/runtime/libtimer_patch.dart:21:15)
#11     _Timer._runTimers (dart:isolate/runtime/libtimer_impl.dart:382:19)
#12     _Timer._handleMessage (dart:isolate/runtime/libtimer_impl.dart:416:5)
#13     _RawReceivePortImpl._handleMessage (dart:isolate/runtime/libisolate_patch.dart:171:12)

--- Re-run this test:
python tools/test.py -m release -n dartk-optcounter-linux-release-x64 service/dominator_tree_vm_test

This test was running with the compactor enabled for all major collections. It's possible the compactor failed to update some pointer.

The text was updated successfully, but these errors were encountered:

rmacnak-google · 2018-11-15T18:36:18Z

Possibly related: same test with compactor encountered an illegal cid in the parallel marker.

FAILED: dartk-vm release_ia32 service/dominator_tree_vm_test
Expected: Pass
Actual: Crash

--- Command "vm" (took 27.000341s):
DART_CONFIGURATION=ReleaseIA32 out/ReleaseIA32/dart --use_compactor --optimization-counter-threshold=5 --ignore-unrecognized-flags --packages=/b/s/w/ir/.packages /b/s/w/ir/runtime/observatory/tests/service/dominator_tree_vm_test.dart

exit code:
-6

stdout:
** Launching SERVICE_TEST_TESTEE=true out/ReleaseIA32/dart -DDART_SERVICE_USE_AUTH=false --profiler --use_compactor --optimization-counter-threshold=5 --ignore-unrecognized-flags --packages=/b/s/w/ir/.packages --enable-vm-service:0 /b/s/w/ir/runtime/observatory/tests/service/dominator_tree_vm_test.dart
>testee>out> Observatory listening on http://127.0.0.1:40473/
** Signaled to run test queries on http://127.0.0.1:40473/
>testee>out> 
Loading VM...
Done loading VM
Running dominator_tree_vm_test.dart [1/1]

stderr:
Class Id: 0
../../runtime/vm/raw_object.cc: 298: error: unreachable code
Dumping native stack trace for thread 3b74
  [0x0108883c] dart::Profiler::DumpStackTrace(void*)
  [0x0108883c] dart::Profiler::DumpStackTrace(void*)
  [0x01286054] Dart_DumpNativeStackTrace
  [0x0128774b] dart::Assert::Fail(char const*, ...)
  [0x0109a07d] dart::RawObject::VisitPointersPredefined(dart::ObjectPointerVisitor*, int)
  [0x0124948f] Unknown symbol
  [0x012493a1] Unknown symbol
  [0x0124972e] Unknown symbol
  [0x011115da] dart::ThreadPool::Worker::Loop()
  [0x0111149e] dart::ThreadPool::Worker::Main(unsigned int)
  [0x01084cf3] Unknown symbol
  [0xf76b3f72] Unknown symbol
-- End of DumpStackTrace

--- Re-run this test:
python tools/test.py -m release -a ia32 -n dartk-optcounter-linux-release-ia32 --write-logs service/dominator_tree_vm_test

(gdb) bt
#0  0xf76e6cd9 in ?? () from /lib/ld-linux.so.2
#1  0x01287753 in dart::Assert::Fail (this=0xf02ff090, format=0x19117 "%s") at ../../runtime/platform/assert.cc:43
#2  0x0109a07d in dart::RawObject::VisitPointersPredefined (this=0xf3719ca1, visitor=0x341e800, class_id=0)
    at ../../runtime/vm/raw_object.cc:298
#3  0x0124948f in dart::RawObject::VisitPointersNonvirtual<dart::MarkingVisitorBase<true> > (this=0xf3719ca1, 
    visitor=0x341e800) at ../../runtime/vm/raw_object.h:605
#4  0x012493a1 in dart::MarkingVisitorBase<true>::DrainMarkingStack (this=0x341e800)
    at ../../runtime/vm/heap/marker.cc:308
#5  0x0124972e in dart::MarkTask::Run (this=0x34242a0) at ../../runtime/vm/heap/marker.cc:589
#6  0x011115da in dart::ThreadPool::Worker::Loop (this=0x3412000) at ../../runtime/vm/thread_pool.cc:381
#7  0x0111149e in dart::ThreadPool::Worker::Main (args=54599680) at ../../runtime/vm/thread_pool.cc:436
#8  0x01084cf3 in dart::ThreadStart (data_ptr=0x3b07520) at ../../runtime/vm/os_thread_linux.cc:131
#9  0xf76b3f72 in pthread_getattr_np () from /lib/i386-linux-gnu/libpthread.so.0
#10 0x00000000 in ?? ()

mraleph · 2018-11-16T17:00:34Z

Another failure of the same test:

FAILED: dartk-vm release_x64 service/dominator_tree_vm_test
Expected: Pass
Actual: Crash

--- Command "vm" (took 10.000699s):
DART_CONFIGURATION=ReleaseX64 out/ReleaseX64/dart --use_compactor --optimization-counter-threshold=5 --ignore-unrecognized-flags --packages=/b/s/w/ir/.packages /b/s/w/ir/runtime/observatory/tests/service/dominator_tree_vm_test.dart

exit code:
-6

stdout:
** Launching SERVICE_TEST_TESTEE=true out/ReleaseX64/dart -DDART_SERVICE_USE_AUTH=false --profiler --use_compactor --optimization-counter-threshold=5 --ignore-unrecognized-flags --packages=/b/s/w/ir/.packages --enable-vm-service:0 /b/s/w/ir/runtime/observatory/tests/service/dominator_tree_vm_test.dart
>testee>out> Observatory listening on http://127.0.0.1:41188/
** Signaled to run test queries on http://127.0.0.1:41188/
>testee>out> 
Loading VM...
Done loading VM
Running dominator_tree_vm_test.dart [1/1]

stderr:
===== CRASH =====
version=2.1.0-edge.e39fa29bd627b9d93fcd01fbc7eb6c757bac53b2 (Thu Nov 15 21:44:03 2018 +0000) on "linux_x64"
si_signo=Segmentation fault(11), si_code=1, si_addr=0x7fad83640020
Dumping native stack trace for thread b1b
  [0x000055e6556836a4] dart::GCCompactor::VisitPointers(dart::RawObject**, dart::RawObject**)
  [0x000055e6556836a4] dart::GCCompactor::VisitPointers(dart::RawObject**, dart::RawObject**)
  [0x000055e655683583] dart::CompactorTask::SlideBlock(unsigned long, dart::ForwardingPage*)
  [0x000055e655682fce] dart::CompactorTask::Run()
  [0x000055e655537cfd] dart::ThreadPool::Worker::Loop()
  [0x000055e655537b94] dart::ThreadPool::Worker::Main(unsigned long)
  [0x000055e65549aad9] Unknown symbol
-- End of DumpStackTrace

--- Re-run this test:
python tools/test.py -m release -n dartk-optcounter-linux-release-x64 --write-logs service/dominator_tree_vm_test

Core dumps are available: https://isolateserver.appspot.com/browse?namespace=default-gzip&hash=8eeb0289b745871e5be9981320d83f4ea9fcc971

dcharkes · 2018-11-19T12:52:20Z

standalone_2/fragmentation_test causes the same crash, see #35210

dcharkes · 2018-11-19T13:31:42Z

#31421 also failed in standalone_2/fragmentation_test and also mentions compaction.

dcharkes · 2018-11-19T13:42:03Z

another stack trace from 12 days ago with a crash in raw_object.cc:298, but this one is without the --use_compactor flag:

FAILED: dartk-vm release_ia32 standalone_2/io/http_outgoing_size_test
Expected: Pass
Actual: Crash

--- Command "vm" (took 05.000854s):
DART_CONFIGURATION=ReleaseIA32 out/ReleaseIA32/dart --short_socket_write --optimization-counter-threshold=5 --sync-async --ignore-unrecognized-flags --packages=/b/s/w/ir/.packages /b/s/w/ir/tests/standalone_2/io/http_outgoing_size_test.dart

exit code:
-6

stderr:
Class Id: 0
../../runtime/vm/raw_object.cc: 298: error: unreachable code
Dumping native stack trace for thread 565d
  [0x010893ec] dart::Profiler::DumpStackTrace(void*)
  [0x010893ec] dart::Profiler::DumpStackTrace(void*)
  [0x012896b4] Dart_DumpNativeStackTrace
  [0x0128adab] dart::Assert::Fail(char const*, ...)
  [0x0109a9e9] dart::RawObject::VisitPointersPredefined(dart::ObjectPointerVisitor*, int)
  [0x01254365] dart::Scavenger::ProcessToSpace(dart::ScavengerVisitor*)
  [0x01254c90] dart::Scavenger::Scavenge()
  [0x01249073] dart::Heap::CollectNewSpaceGarbage(dart::Thread*, dart::Heap::GCReason)
  [0x01247bff] dart::Heap::AllocateOld(int, dart::HeapPage::PageType)
  [0x00ffba6e] dart::Object::Allocate(int, int, dart::Heap::Space)
  [0x01011d67] dart::Function::New(dart::String const&, dart::RawFunction::Kind, bool, bool, bool, bool, bool, dart::Object const&, dart::TokenPosition, dart::Heap::Space)
  [0x0102c28b] dart::Function::NewSignatureFunction(dart::Object const&, dart::Function const&, dart::TokenPosition, dart::Heap::Space)
  [0x01212576] dart::kernel::TypeTranslator::BuildFunctionType(bool)
  [0x01211ed9] dart::kernel::TypeTranslator::BuildTypeInternal(bool)
  [0x0121212f] dart::kernel::TypeTranslator::BuildVariableType()
  [0x0121b455] dart::kernel::ScopeBuilder::AddVariableDeclarationParameter(int, dart::kernel::ScopeBuilder::ParameterTypeCheckMode, dart::kernel::ProcedureAttributesMetadata const&)
  [0x0121719c] dart::kernel::ScopeBuilder::BuildScopes()
  [0x01086660] dart::ParsedFunction::EnsureKernelScopes()
  [0x011eb63e] dart::kernel::StreamingFlowGraphBuilder::BuildGraph()
  [0x01200786] dart::kernel::FlowGraphBuilder::BuildGraph()
  [0x0122e15c] dart::DartCompilationPipeline::BuildFlowGraph(dart::Zone*, dart::ParsedFunction*, dart::ZoneGrowableArray<dart::ICData const*>*, int, bool)
  [0x01230ae9] dart::CompileParsedFunctionHelper::Compile(dart::CompilationPipeline*)
  [0x01231b90] Unknown symbol
  [0x01232a08] dart::Compiler::CompileOptimizedFunction(dart::Thread*, dart::Function const&, int)
  [0x012344a8] dart::BackgroundCompiler::Run()
  [0x01234bd7] Unknown symbol
  [0x01118ada] dart::ThreadPool::Worker::Loop()
  [0x0111899e] dart::ThreadPool::Worker::Main(unsigned int)
  [0x010858a3] Unknown symbol
  [0xf7740f72] Unknown symbol
-- End of DumpStackTrace

--- Re-run this test:
python tools/test.py -m release -a ia32 -n dartk-optcounter-linux-release-ia32 standalone_2/io/http_outgoing_size_test

a-siva · 2018-11-20T18:15:23Z

I saw a similar crash in one of the configurations when I was submitting a CL:
https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket.appspot.com/8929319077009299600/+/steps/vm_tests_shard_5/0/stdout

The stack trace

>testee>err> ../../runtime/vm/raw_object.cc: 298: error: unreachable code
>testee>err> Dumping native stack trace for thread 7f4e
>testee>err>   [0x010881cc] dart::Profiler::DumpStackTrace(void*)
>testee>err>   [0x010881cc] dart::Profiler::DumpStackTrace(void*)
>testee>err>   [0x01282ed4] Dart_DumpNativeStackTrace
>testee>err>   [0x012845eb] dart::Assert::Fail(char const*, ...)
>testee>err>   [0x01099a06] dart::RawObject::VisitPointersPredefined(dart::ObjectPointerVisitor*, int)
>testee>err>   [0x0124639f] Unknown symbol
>testee>err>   [0x012462b1] Unknown symbol
>testee>err>   [0x0124663e] Unknown symbol
>testee>err>   [0x01110c5a] dart::ThreadPool::Worker::Loop()
>testee>err>   [0x01110b1e] dart::ThreadPool::Worker::Main(unsigned int)
>testee>err>   [0x01084693] Unknown symbol
>testee>err>   [0xf777ef72] Unknown symbol
>testee>err> -- End of DumpStackTrace

StubEntry was holding raw pointers across safepoints in several places in the compiler. Bug: #34934 Change-Id: Id17cb7651f4d3508915c8cb90f5cb2353df6dac0 Reviewed-on: https://dart-review.googlesource.com/c/85040 Reviewed-by: Régis Crelier <regis@google.com> Commit-Queue: Ryan Macnak <rmacnak@google.com>

Bug: #34934 Change-Id: Ibba2ae5837985a32bd84da4b33bd091dcec3427d Reviewed-on: https://dart-review.googlesource.com/c/85141 Reviewed-by: Régis Crelier <regis@google.com> Reviewed-by: Alexander Markov <alexmarkov@google.com> Commit-Queue: Ryan Macnak <rmacnak@google.com>

rmacnak-google · 2018-11-28T00:01:03Z

I haven't seen this fail since the StubEntry change, so I'll assume that's what the bots were encountering.

StubEntry was holding raw pointers across safepoints in several places in the compiler. Bug: dart-lang#34934 Change-Id: Id17cb7651f4d3508915c8cb90f5cb2353df6dac0 Reviewed-on: https://dart-review.googlesource.com/c/85040 Reviewed-by: Régis Crelier <regis@google.com> Commit-Queue: Ryan Macnak <rmacnak@google.com>

rmacnak-google added area-vm Use area-vm for VM related issues, including code coverage, and the AOT and JIT backends. gardening labels Oct 25, 2018

dcharkes mentioned this issue Nov 19, 2018

Crashes with compactor and optimization-counter #35210

Closed

a-siva added the crash Process exits with SIGSEGV, SIGABRT, etc. An unhandled exception is not a crash. label Nov 20, 2018

rmacnak-google mentioned this issue Nov 21, 2018

standalone_2/fragmentation_test crash on bots; not reproducible locally #31421

Closed

rmacnak-google closed this as completed Nov 28, 2018

mraleph mentioned this issue Dec 12, 2018

Segfault (in MarkingVisitorBase<true>::VisitPointers?) in Flutter etst run on Chromium macOS #35388

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dangling pointer observed in service heap snapshot test - possible compactor bug #34934

Dangling pointer observed in service heap snapshot test - possible compactor bug #34934

rmacnak-google commented Oct 25, 2018

rmacnak-google commented Nov 15, 2018

mraleph commented Nov 16, 2018

dcharkes commented Nov 19, 2018

dcharkes commented Nov 19, 2018

dcharkes commented Nov 19, 2018

a-siva commented Nov 20, 2018

rmacnak-google commented Nov 28, 2018

Dangling pointer observed in service heap snapshot test - possible compactor bug #34934

Dangling pointer observed in service heap snapshot test - possible compactor bug #34934

Comments

rmacnak-google commented Oct 25, 2018

rmacnak-google commented Nov 15, 2018

mraleph commented Nov 16, 2018

dcharkes commented Nov 19, 2018

dcharkes commented Nov 19, 2018

dcharkes commented Nov 19, 2018

a-siva commented Nov 20, 2018

rmacnak-google commented Nov 28, 2018