Akka.Persistence.ConfigSpec: unable to find akka.persistence.dispatchers.default-stream-dispatcher

[Aaronontheweb]

Detected a real bug here, I think related to some of the previous HOCON changes:

Akka.Persistence.Tests.PersistenceConfigSpec.Persistence_should_be_able_to_register_the_same_snapshot_store_under_different_paths

[ERROR][2/19/2020 5:17:50 PM][Thread 0035][akka://AkkaSpec/system/akka.persistence.snapshot-store.test2] Error while creating actor instance of type Akka.Persistence.Tests.PersistenceConfigSpec+TestSnapshotStore with 1 args: ({
  class : "Akka.Persistence.Tests.PersistenceConfigSpec+TestSnapshotStore, Akka.Persistence.Tests",
  plugin-dispatcher : "akka.actor.default-dispatcher",
  test-value : "D"
})
Cause: [akka://AkkaSpec/system/akka.persistence.snapshot-store.test2#413508506]: Akka.Actor.ActorInitializationException: Exception during creation
 ---> System.TypeLoadException: Error while creating actor instance of type Akka.Persistence.Tests.PersistenceConfigSpec+TestSnapshotStore with 1 args: ({
  class : "Akka.Persistence.Tests.PersistenceConfigSpec+TestSnapshotStore, Akka.Persistence.Tests",
  plugin-dispatcher : "akka.actor.default-dispatcher",
  test-value : "D"
})
 ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> Hocon.ConfigurationException: Dispatcher akka.persistence.dispatchers.default-stream-dispatcher not configured.
   at Akka.Dispatch.Dispatchers.LookupConfigurator(String id) in /home/vsts/work/1/s/src/core/Akka/Dispatch/Dispatchers.cs:line 312
   at Akka.Dispatch.Dispatchers.Lookup(String dispatcherName) in /home/vsts/work/1/s/src/core/Akka/Dispatch/Dispatchers.cs:line 286
   at Akka.Persistence.Snapshot.LocalSnapshotStore..ctor() in /home/vsts/work/1/s/src/core/Akka.Persistence/Snapshot/LocalSnapshotStore.cs:line 56
   at Akka.Persistence.Tests.PersistenceConfigSpec.TestSnapshotStore..ctor(Config config) in /home/vsts/work/1/s/src/core/Akka.Persistence.Tests/PersistenceConfigSpec.cs:line 57
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture)
   at Akka.Actor.Props.ActivatorProducer.Produce() in /home/vsts/work/1/s/src/core/Akka/Actor/Props.cs:line 639
   at Akka.Actor.Props.NewActor() in /home/vsts/work/1/s/src/core/Akka/Actor/Props.cs:line 577
   --- End of inner exception stack trace ---
   at Akka.Actor.Props.NewActor() in /home/vsts/work/1/s/src/core/Akka/Actor/Props.cs:line 577
   at Akka.Actor.ActorCell.CreateNewActorInstance() in /home/vsts/work/1/s/src/core/Akka/Actor/ActorCell.cs:line 361
   at Akka.Actor.ActorCell.<>c__DisplayClass117_0.<NewActor>b__0() in /home/vsts/work/1/s/src/core/Akka/Actor/ActorCell.cs:line 340
   at Akka.Actor.ActorCell.UseThreadContext(Action action) in /home/vsts/work/1/s/src/core/Akka/Actor/ActorCell.cs:line 381
   at Akka.Actor.ActorCell.NewActor() in /home/vsts/work/1/s/src/core/Akka/Actor/ActorCell.cs:line 341
   at Akka.Actor.ActorCell.Create(Exception failure) in /home/vsts/work/1/s/src/core/Akka/Actor/ActorCell.DefaultMessages.cs:line 473
   --- End of inner exception stack trace ---
   at Akka.Actor.ActorCell.Create(Exception failure) in /home/vsts/work/1/s/src/core/Akka/Actor/ActorCell.DefaultMessages.cs:line 473
   at Akka.Actor.ActorCell.SysMsgInvokeAll(EarliestFirstSystemMessageList messages, Int32 currentState) in /home/vsts/work/1/s/src/core/Akka/Actor/ActorCell.DefaultMessages.cs:line 271
[INFO][2/19/2020 5:17:50 PM][Thread 0035][akka://AkkaSpec/system/akka.persistence.snapshot-store.test2] Message [TestRequest] from akka://AkkaSpec/system/testActor254 to akka://AkkaSpec/system/akka.persistence.snapshot-store.test2 was not delivered. [1] dead letters encountered .This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[DEBUG][2/19/2020 5:17:53 PM][Thread 0013][ActorSystem(AkkaSpec)] System shutdown initiated
[DEBUG][2/19/2020 5:17:53 PM][Thread 0035][EventStream] Shutting down: StandardOutLogger started
[DEBUG][2/19/2020 5:17:53 PM][Thread 0035][EventStream] All default loggers stopped

Originally posted by @Aaronontheweb in #4211 (comment)

[Aaronontheweb]

Appears to be a race condition - an instance where the default Akka.Persistence settings aren't applied before the LocalSnapshotStore attempts to consume them.

[Aaronontheweb]

Looks like the issue is that Hocon.Config.Root.TryGetObject threw an exception:

https://github.com/akkadotnet/HOCON/blob/9d176be55aa9ef3a20df5db09e94a4c9c22fe99f/src/Hocon/Impl/HoconValue.cs#L69-L91

System.ArgumentNullException: 'Value cannot be null. Parameter name: source'

[Aaronontheweb]

cc @Arkatufus output from a DumpConfig call at the time the exception was thrown:

dumpconfig.txt