Pooled objects unexpectedly migrate between scenes

[IlyaKvant]

General
Unity version: 2021.3.18f1
Fish-Networking version: 4.5.4hf0 Pro
Discord link: https://discord.com/channels/424284635074134018/1304433019167313992
Related: #812
Package: Unexpected.Migration.081124.unitypackage.zip

Description
In the example, the server and client load the global game scene.
After the client initializes, both start spawning SimpleBullet every 300ms.
The SimpleBullet is a pooled object that allows for predicting spawn/despawn.
The blue bullet represents is own object, while the grey one represents the other player.

When we go to another global scene you can see bullets that shouldn't be there. We can also change the scene again, and the bullets will still be there.
At the same time, there are no bullets on the server, there is no script that would create them either, and everyone has SceneCondition.

The SceneCondition condition is set in ObserverManager.
All NetworkObjects have a NetworkObserver component with the use Manager parameter.
The spawn logic is the same as here: #812

Replication
Steps to reproduce the behaviour:

1. Open the "NestedSpawn.Menu" scene in both windows;
2. Start the Server and Client in one window and Client in other;
3. Press the "Load Scene" button on Server;
4. Wait for a few bullets to be re-spawned;
5. Click to the "Next scene" button;
6. If no bullets, repeat the loop;

Expected behaviour
Bullets should only be on their own scene and should not migrate.
It seems that there should be no warnings and errors at all, as there is no logic other than spawn and despawn.

Errors

Exception

Destroying assets is not permitted to avoid data loss.
If you really want to remove an asset use DestroyImmediate (theObject, true);
UnityEngine.Object:Destroy (UnityEngine.Object)
FishNet.Observing.NetworkObserver:Deinitialize (bool) (at Assets/FishNet/Runtime/Observing/NetworkObserver.cs:159)
FishNet.Object.NetworkObject:OnDestroy () (at Assets/FishNet/Runtime/Object/NetworkObject.cs:479)

Warnings

Action cannot be completed as object Bullet(Clone) [Id 228] is not spawned.
UnityEngine.Debug:LogWarning (object)
FishNet.Managing.Logging.LevelLoggingConfiguration:LogWarning (string) (at Assets/FishNet/Runtime/Managing/Logging/LevelLoggingConfiguration.cs:119)
FishNet.Managing.NetworkManager:InternalLogWarning (string) (at Assets/FishNet/Runtime/Managing/NetworkManager.Logging.cs:75)
FishNet.Managing.NetworkManagerExtensions:LogWarning (FishNet.Managing.NetworkManager,string) (at Assets/FishNet/Runtime/Managing/NetworkManager.Logging.cs:134)
FishNet.Object.NetworkBehaviour:IsSpawnedWithWarning () (at Assets/FishNet/Runtime/Object/NetworkBehaviour.RPCs.cs:374)
FishNet.Object.NetworkBehaviour:SendServerRpc (uint,FishNet.Serializing.PooledWriter,FishNet.Transporting.Channel,FishNet.Object.DataOrderType) (at Assets/FishNet/Runtime/Object/NetworkBehaviour.RPCs.cs:235)
FishNet.Component.Transforming.NetworkTransform:RpcWriter___Server_ServerUpdateTransform_2713644489 (System.ArraySegment`1<byte>,FishNet.Transporting.Channel)
FishNet.Component.Transforming.NetworkTransform:ServerUpdateTransform (System.ArraySegment`1<byte>,FishNet.Transporting.Channel)
FishNet.Component.Transforming.NetworkTransform:SendToServer (FishNet.Component.Transforming.NetworkTransform/TransformData) (at Assets/FishNet/Runtime/Generated/Component/NetworkTransform/NetworkTransform.cs:1754)
FishNet.Component.Transforming.NetworkTransform:TimeManager_OnPostTick () (at Assets/FishNet/Runtime/Generated/Component/NetworkTransform/NetworkTransform.cs:906)
FishNet.Managing.Timing.TimeManager:IncreaseTick () (at Assets/FishNet/Runtime/Managing/Timing/TimeManager.cs:715)
FishNet.Managing.Timing.TimeManager:<TickUpdate>g__MethodLogic|100_0 () (at Assets/FishNet/Runtime/Managing/Timing/TimeManager.cs:376)
FishNet.Managing.Timing.TimeManager:TickUpdate () (at Assets/FishNet/Runtime/Managing/Timing/TimeManager.cs:366)
FishNet.Transporting.NetworkReaderLoop:Update () (at Assets/FishNet/Runtime/Transporting/NetworkReaderLoop.cs:28)

ObjectId 332 could not be found in Server spawned, nor Server pending despawn. This may occur as clientHost when objects are destroyed before the client receives a despawn packet. In most cases this may be ignored.
UnityEngine.Debug:LogWarning (object)
FishNet.Managing.Logging.LevelLoggingConfiguration:LogWarning (string) (at Assets/FishNet/Runtime/Managing/Logging/LevelLoggingConfiguration.cs:119)
FishNet.Managing.NetworkManager:InternalLogWarning (string) (at Assets/FishNet/Runtime/Managing/NetworkManager.Logging.cs:75)
FishNet.Managing.NetworkManagerExtensions:LogWarning (FishNet.Managing.NetworkManager,string) (at Assets/FishNet/Runtime/Managing/NetworkManager.Logging.cs:134)
FishNet.Managing.Client.ClientObjects:GetInstantiatedNetworkObject (FishNet.Managing.Client.CachedNetworkObject) (at Assets/FishNet/Runtime/Managing/Client/Object/ClientObjects.cs:647)
FishNet.Managing.Client.ClientObjectCache:<Iterate>g__ProcessObject|13_0 (FishNet.Managing.Client.CachedNetworkObject,bool,int,FishNet.Managing.Client.ClientObjectCache/<>c__DisplayClass13_0&) (at Assets/FishNet/Runtime/Managing/Client/Object/ObjectCaching.cs:296)
FishNet.Managing.Client.ClientObjectCache:Iterate () (at Assets/FishNet/Runtime/Managing/Client/Object/ObjectCaching.cs:270)
FishNet.Managing.Client.ClientObjects:IterateObjectCache () (at Assets/FishNet/Runtime/Managing/Client/Object/ClientObjects.cs:490)
FishNet.Managing.Client.ClientManager:ParseReader (FishNet.Serializing.PooledReader,FishNet.Transporting.Channel,bool) (at Assets/FishNet/Runtime/Managing/Client/ClientManager.cs:538)
FishNet.Managing.Client.ClientManager:ParseReceived (FishNet.Transporting.ClientReceivedDataArgs) (at Assets/FishNet/Runtime/Managing/Client/ClientManager.cs:385)
FishNet.Managing.Client.ClientManager:Transport_OnClientReceivedData (FishNet.Transporting.ClientReceivedDataArgs) (at Assets/FishNet/Runtime/Managing/Client/ClientManager.cs:343)
FishNet.Transporting.Tugboat.Tugboat:HandleClientReceivedDataArgs (FishNet.Transporting.ClientReceivedDataArgs) (at Assets/FishNet/Runtime/Transporting/Transports/Tugboat/Tugboat.cs:245)
FishNet.Transporting.Tugboat.Client.ClientSocket:IterateIncoming () (at Assets/FishNet/Runtime/Transporting/Transports/Tugboat/Core/ClientSocket.cs:295)
FishNet.Transporting.Tugboat.Tugboat:IterateIncoming (bool) (at Assets/FishNet/Runtime/Transporting/Transports/Tugboat/Tugboat.cs:218)
FishNet.Managing.Transporting.TransportManager:IterateIncoming (bool) (at Assets/FishNet/Runtime/Managing/Transporting/TransportManager.cs:705)
FishNet.Managing.Timing.TimeManager:TryIterateData (bool) (at Assets/FishNet/Runtime/Managing/Timing/TimeManager.cs:1057)
FishNet.Managing.Timing.TimeManager:IncreaseTick () (at Assets/FishNet/Runtime/Managing/Timing/TimeManager.cs:699)
FishNet.Managing.Timing.TimeManager:<TickUpdate>g__MethodLogic|100_0 () (at Assets/FishNet/Runtime/Managing/Timing/TimeManager.cs:376)
FishNet.Managing.Timing.TimeManager:TickUpdate () (at Assets/FishNet/Runtime/Managing/Timing/TimeManager.cs:366)
FishNet.Transporting.NetworkReaderLoop:Update () (at Assets/FishNet/Runtime/Transporting/NetworkReaderLoop.cs:28)

[FirstGearGames]

Just a heads up, marking as waiting on information until we get more answers in Discord.

[FirstGearGames]

Closed until investigated on Discord.