Flaky assert hit in canonicalization verifier on isolate shutdown

[mkustermann]

From this log:

/=======================================================================================================\
| dartk-weak-asserts-linux-debug-x64:lib/isolate/issue_6610_test/0 broke (Pass -> Crash, expected Pass) |
\=======================================================================================================/

--- Command "vm" (took 03.000964s):
DART_CONFIGURATION=DebugX64 out/DebugX64/dart --enable_asserts --enable-isolate-groups --no-sound-null-safety --ignore-unrecognized-flags --packages=/b/s/w/ir/.packages /b/s/w/ir/tests/lib/isolate/issue_6610_test.dart

exit code:
-6

stderr:
../../runtime/vm/heap/safepoint.h: 301: error: expected: !T->isolate_group()->safepoint_handler()->IsOwnedByTheThread(T)
version=2.14.0-edge.e84240355f79f8fd07e1d697500a9c4c16749867 (be) (Thu Jul 22 20:38:25 2021 +0000) on "linux_x64"
pid=22416, thread=22503, isolate_group=main(0x55737fa29800), isolate=leaf(0x55737fc89000)
isolate_instructions=55737e047d60, vm_instructions=55737e047d60
  pc 0x000055737e428e5c fp 0x00007efcf5ffe770 dart::Profiler::DumpStackTrace(void*)+0x7c
  pc 0x000055737e047f14 fp 0x00007efcf5ffe850 dart::Assert::Fail(char const*, ...)+0x84
  pc 0x000055737e314461 fp 0x00007efcf5ffe880 out/DebugX64/dart+0x2140461
  pc 0x000055737e314266 fp 0x00007efcf5ffe8d0 dart::SafepointMutexLocker::SafepointMutexLocker(dart::ThreadState*, dart::Mutex*)+0x86
  pc 0x000055737e3c38e1 fp 0x00007efcf5ffe970 dart::TypeParameter::CheckIsCanonical(dart::Thread*) const+0x81
  pc 0x000055737e540ce0 fp 0x00007efcf5ffe9b0 dart::VerifyCanonicalVisitor::VisitObject(dart::ObjectPtr)+0x80
  pc 0x000055737e528ff5 fp 0x00007efcf5ffea10 dart::OldPage::VisitObjects(dart::ObjectVisitor*) const+0xf5
  pc 0x000055737e52cc8d fp 0x00007efcf5ffea90 dart::PageSpace::VisitObjects(dart::ObjectVisitor*) const+0x5d
  pc 0x000055737e2d401a fp 0x00007efcf5ffeb30 dart::IsolateGroup::ValidateConstants()+0xfa
  pc 0x000055737e95255c fp 0x00007efcf5ffebb0 Dart_ShutdownIsolate+0x10c
  pc 0x000055737e31b5d7 fp 0x00007efcf5ffec30 dart::MessageHandler::TaskCallback()+0x607
  pc 0x000055737e4f308b fp 0x00007efcf5ffecd0 dart::ThreadPool::WorkerLoop(dart::ThreadPool::Worker*)+0x1fb
  pc 0x000055737e4f3a27 fp 0x00007efcf5ffed20 dart::ThreadPool::Worker::Main(unsigned long)+0x147
  pc 0x000055737e420574 fp 0x00007efcf5ffee70 out/DebugX64/dart+0x224c574
-- End of DumpStackTrace

--- Re-run this test:
python3 tools/test.py -n dartk-weak-asserts-linux-debug-x64 lib/isolate/issue_6610_test/0

/cc @rmacnak-google @aam

[aam]

Canonical verification code is intrisically racy as it runs under safepoint https://github.com/dart-lang/sdk/blob/master/runtime/vm/isolate.cc#L993 and attempts to grab
SafepointMutexLocker ml(thread->isolate_group()->constant_canonicalization_mutex()); at
https://github.com/dart-lang/sdk/blob/master/runtime/vm/object.cc#L19088 which leads to problems as this mutex might be held by some other thread that was forced to a safepoint already.