I guess you should use addFactoryDefinition and then ->getResultDefition()->setType(...).
FactoryDefinition is for factories generated from interface, while setFactory() define how is instance created (@foo::getInstance() for $this->getService('foo')->getInstance() or Foo::class for new Foo), these things are not related.
InjectExtension not work when set factory and type
If you put definition into neon with factory and type keys defined - is it working? Imho it's problem with priorities, not with extension itself.