Infinite recursion in type finalization

[mkustermann]

Reproduction

abstract class Base<A, B> {}

extension on Object {
  B? foo<A extends Base<A, B>, B extends Base<A, B>>(B? orig) {
    return null;
  }
}

main() {
  print(Object().foo);
}

causes the VM to enter into a stack overflow in C++ due to infinite recursion:

#62804 0x00005555574dd4e2 in main (argc=2, argv=0x7fffffffbe48) at ../../runtime/bin/main.cc:1407
#62803 0x00005555574dc9e1 in dart::bin::main (argc=2, argv=0x7fffffffbe48) at ../../runtime/bin/main.cc:1369
#62802 0x00005555574db5ec in dart::bin::RunMainIsolate (script_name=0x555558b36010 "test.dart", package_config_override=0x0, dart_options=0x7fffffffbcd8) at ../../runtime/bin/main.cc:1041
#62801 0x000055555804b16b in Dart_GetField (container=0x555558baad18, name=0x555558baad20) at ../../runtime/vm/dart_api_impl.cc:4805
#62800 0x000055555792db7b in dart::Library::InvokeGetter (this=0x555558bab390, getter_name=..., throw_nsm_if_absent=true, respect_reflectable=false, check_is_entrypoint=false) at ../../runtime/vm/object.cc:13180
#62799 0x000055555792c0fd in dart::Library::LookupLocalOrReExportObject (this=0x555558bab390, name=...) at ../../runtime/vm/object.cc:12712
#62798 0x0000555557929906 in dart::Library::EnsureTopLevelClassIsFinalized (this=0x555558bab390) at ../../runtime/vm/object.cc:12699
#62797 0x00005555578f878a in dart::Class::EnsureIsFinalized (this=0x555558bab3a0, thread=0x55555949c400) at ../../runtime/vm/object.cc:4436
#62796 0x000055555778b227 in dart::ClassFinalizer::LoadClassMembers (cls=...) at ../../runtime/vm/class_finalizer.cc:1212
#62795 0x00005555577899d6 in dart::ClassFinalizer::FinalizeClass (cls=...) at ../../runtime/vm/class_finalizer.cc:1134
#62794 0x00005555577890b8 in dart::ClassFinalizer::FinalizeMemberTypes (cls=...) at ../../runtime/vm/class_finalizer.cc:978
#62793 0x0000555557787aa1 in dart::ClassFinalizer::FinalizeType (type=..., finalization=dart::ClassFinalizer::kCanonicalize, pending_types=0x0) at ../../runtime/vm/class_finalizer.cc:814
#62792 0x0000555557788b87 in dart::ClassFinalizer::FinalizeSignature (zone=0x555558baaf30, signature=..., finalization=dart::ClassFinalizer::kCanonicalize, pending_types=0x0) at ../../runtime/vm/class_finalizer.cc:876
#62791 0x0000555557787aa1 in dart::ClassFinalizer::FinalizeType (type=..., finalization=dart::ClassFinalizer::kFinalize, pending_types=0x0) at ../../runtime/vm/class_finalizer.cc:814
#62790 0x0000555557788d39 in dart::ClassFinalizer::FinalizeSignature (zone=0x555558baaf30, signature=..., finalization=dart::ClassFinalizer::kFinalize, pending_types=0x0) at ../../runtime/vm/class_finalizer.cc:894
#62789 0x00005555579629d7 in dart::FunctionType::SetIsFinalized (this=0x555559365cc0) at ../../runtime/vm/object.cc:20354
#62788 0x000055555791bb1e in dart::FunctionType::IsInstantiated (this=0x555559365cc0, genericity=dart::kAny, num_free_fun_type_params=0, trail=0x0) at ../../runtime/vm/object.cc:9422
#62787 0x0000555557968573 in dart::TypeParameter::IsInstantiated (this=0x5555593657f0, genericity=dart::kAny, num_free_fun_type_params=0, trail=0x0) at ../../runtime/vm/object.cc:21491
#62786 0x0000555557962e4e in dart::Type::IsInstantiated (this=0x555559365800, genericity=dart::kAny, num_free_fun_type_params=0, trail=0x0) at ../../runtime/vm/object.cc:20460
#62785 0x000055555790ee2a in dart::TypeArguments::IsSubvectorInstantiated (this=0x555559365810, from_index=0, len=2, genericity=dart::kAny, num_free_fun_type_params=0, trail=0x0) at ../../runtime/vm/object.cc:6609
#62784 0x0000555557968573 in dart::TypeParameter::IsInstantiated (this=0x555559365830, genericity=dart::kAny, num_free_fun_type_params=0, trail=0x0) at ../../runtime/vm/object.cc:21491
#62783 0x00005555579676a9 in dart::TypeRef::IsInstantiated (this=0x555559365840, genericity=dart::kAny, num_free_fun_type_params=0, trail=0x7ffff742e2f0) at ../../runtime/vm/object.cc:21284
#62782 0x0000555557962e4e in dart::Type::IsInstantiated (this=0x555559365850, genericity=dart::kAny, num_free_fun_type_params=0, trail=0x7ffff742e2f0) at ../../runtime/vm/object.cc:20460
#62781 0x000055555790ee2a in dart::TypeArguments::IsSubvectorInstantiated (this=0x555559365860, from_index=0, len=2, genericity=dart::kAny, num_free_fun_type_params=0, trail=0x7ffff742e2f0) at ../../runtime/vm/object.cc:6609
#62780 0x0000555557968573 in dart::TypeParameter::IsInstantiated (this=0x555559365880, genericity=dart::kAny, num_free_fun_type_params=0, trail=0x7ffff742e2f0) at ../../runtime/vm/object.cc:21491
#62779 0x0000555557962e4e in dart::Type::IsInstantiated (this=0x5555593658a0, genericity=dart::kAny, num_free_fun_type_params=0, trail=0x7ffff742e2f0) at ../../runtime/vm/object.cc:20460
#62778 0x000055555790ee2a in dart::TypeArguments::IsSubvectorInstantiated (this=0x5555593658b0, from_index=0, len=2, genericity=dart::kAny, num_free_fun_type_params=0, trail=0x7ffff742e2f0) at ../../runtime/vm/object.cc:6609
#62777 0x0000555557968573 in dart::TypeParameter::IsInstantiated (this=0x5555593658d0, genericity=dart::kAny, num_free_fun_type_params=0, trail=0x7ffff742e2f0) at ../../runtime/vm/object.cc:21491
...
#13 0x0000555557962e4e in dart::Type::IsInstantiated (this=0x555559014040, genericity=dart::kAny, num_free_fun_type_params=0, trail=0x7ffff742e2f0) at ../../runtime/vm/object.cc:20460
#12 0x000055555790ee2a in dart::TypeArguments::IsSubvectorInstantiated (this=0x555559014050, from_index=0, len=2, genericity=dart::kAny, num_free_fun_type_params=0, trail=0x7ffff742e2f0) at ../../runtime/vm/object.cc:6609
#11 0x0000555557968573 in dart::TypeParameter::IsInstantiated (this=0x555559014070, genericity=dart::kAny, num_free_fun_type_params=0, trail=0x7ffff742e2f0) at ../../runtime/vm/object.cc:21491
#10 0x0000555557962e4e in dart::Type::IsInstantiated (this=0x555559014090, genericity=dart::kAny, num_free_fun_type_params=0, trail=0x7ffff742e2f0) at ../../runtime/vm/object.cc:20460
#9  0x000055555790ee2a in dart::TypeArguments::IsSubvectorInstantiated (this=0x5555590140a0, from_index=0, len=2, genericity=dart::kAny, num_free_fun_type_params=0, trail=0x7ffff742e2f0) at ../../runtime/vm/object.cc:6609
#8  0x0000555557968557 in dart::TypeParameter::IsInstantiated (this=0x5555590140c0, genericity=dart::kAny, num_free_fun_type_params=0, trail=0x7ffff742e2f0) at ../../runtime/vm/object.cc:21490
#7  0x000055555771d9fd in dart::AbstractType::Handle (ptr=...) at ../../runtime/vm/object.h:8129
#6  0x000055555771de02 in dart::AbstractType::HandleImpl (zone=0x555558baaf30, ptr=...) at ../../runtime/vm/object.h:8129
#5  dart::AbstractType::initializeHandle (obj=0x5555590140d0, ptr=...) at ../../runtime/vm/object.h:8129
#4  0x000055555771da91 in dart::Object::SetPtr (this=0x5555590140d0, value=...) at ../../runtime/vm/object.h:11585
#3  0x000055555770c29a in dart::NoSafepointScope::NoSafepointScope (this=0x7fffff7ff1a8, thread=0x0) at ../../runtime/vm/thread.h:1282
#2  0x000055555770c5f1 in dart::ThreadStackResource::ThreadStackResource (this=0x7fffff7ff1a8, T=0x55555949c400) at ../../runtime/vm/thread_stack_resource.h:21
#1  0x000055555770c71b in dart::StackResource::StackResource (this=0x7fffff7ff1a8, thread=0x55555949c400) at ../../runtime/vm/allocation.h:26
#0  0x0000555557781808 in dart::StackResource::Init (this=0x7fffff7ff1a8, thread=0x55555949c400) at ../../runtime/vm/allocation.cc:50

/cc @crelier Could you take a look?

[eernstg]

Maybe it's A extends Base<A, B> rather than M extends Base<A, B>?

[mkustermann]

@eernstg Yes - edited the repro example

[crelier]

Thanks for the report and repro!
I started working on a fix in https://dart-review.googlesource.com/c/sdk/+/202820.