NRE from nested NetworkObjects

[IlyaKvant]

General
Unity version: 2021.3.18f1
Fish-Networking version: 4.5.4hf0 Pro
Discord link: https://discord.com/channels/424284635074134018/1293524096423825439 (comments)
Package: NRE.Condition.Report.081124.unitypackage.zip (from #812)

The object hierarchy and spawn logic is the same as here: #796

The SceneCondition condition is set in ObserverManager.
All NetworkObjects have a NetworkObserver component with the use Manager parameter.

Description
When switching from a scene on which nested NetworkObjects were created - an error occurs.

Replication
Steps to reproduce the behaviour:

1. Open the "NestedSpawn.Menu" scene in both windows;
2. Start the Server and Client in one window and Client in other;
3. Press the "Load Scene" button on Server;
4. Wait for the tower with rotating cubes to appear on the server and client;
5. Click to the "Next scene" button - see the error.

Expected behavior
No NRE and a correct transition to the next scene.

Server error 1 (NRE NetworkObject.OnDestroy)

NullReferenceException: Object reference not set to an instance of an object
FishNet.Object.NetworkObject.OnDestroy () (at Assets/FishNet/Runtime/Object/NetworkObject.cs:519)