Service Fabric Actor notification fails to trigger event handler after first invocation due to possible blocking

[dasiths]

I've got 2 Reliable actors called GameActor and PlayerActor. The ClientApp send a message to the PlayerActor when the player makes a move. Then the PlayerActor sends a message to the GameActor to indicate a movement was made. Upon being invoked, the method in the GameActor fires a notification. This notification gets handled by the ClientApp GameEventsHandler. The ClientApp then calls a method on the GameActor to retrieve the latest player positions.

ClientApp -> PlayerActor.MoveTo() -> GameActor.NotifyPlayerMoved() ->
Fire ScoreBoardUpdated event

GameEventsHandler triggered by that event ->
GameActor.GetLatestPlayerInfo()

The problem I'm having is this. The very first time I run it, the GameEventsHandler gets triggered and it tries to call the GameActor as expected. The GameActor receives the message and returns the response expected. But the client doesn't seem to receive the message. It looks like it's blocked as it doesn't throw and error or any output. Any subsequent notifications don't get handled by the event handler at all.

GameActor

        public async Task<IList<PlayerInfo>> GetLatestPlayerInfoAsync(CancellationToken cancellationToken)
        {
            var allPlayers = await StateManager.GetStateAsync<List<string>>("players", cancellationToken);

            var tasks = allPlayers.Select(actorName =>
            {
                var playerActor = ActorProxy.Create<IPlayerActor>(new ActorId(actorName), new Uri(PlayerActorUri));
                return playerActor.GetLatestInfoAsync(cancellationToken);
            }).ToList();

            await Task.WhenAll(tasks);

            return tasks
                .Select(t => t.Result)
                .ToList();
        }

        public async Task NotifyPlayerMovedAsync(PlayerInfo lastMovement, CancellationToken cancellationToken)
        {
            var ev = GetEvent<IGameEvents>();
            ev.ScoreboardUpdated(lastMovement);
        }

PlayerActor

        public async Task MoveToAsync(int x, int y, CancellationToken cancellationToken)
        {
            var playerName = await StateManager.GetStateAsync<string>("playerName", cancellationToken);
            var playerInfo = new PlayerInfo()
            {
                LastUpdate = DateTimeOffset.Now,
                PlayerName = playerName,
                XCoordinate = x,
                YCoordinate = y
            };

            await StateManager.AddOrUpdateStateAsync("positions", new List<PlayerInfo>() { playerInfo }, (key, value) =>
            {
                value.Add(playerInfo);
                return value;
            }, cancellationToken);

            var gameName = await StateManager.GetStateAsync<string>("gameName", cancellationToken);
            var gameActor = ActorProxy.Create<IGameActor>(new ActorId(gameName), new Uri(GameActorUri));
            await gameActor.NotifyPlayerMovedAsync(playerInfo, cancellationToken);
        }

        public async Task<PlayerInfo> GetLatestInfoAsync(CancellationToken cancellationToken)
        {
            var positions = await StateManager.GetStateAsync<List<PlayerInfo>>("positions", cancellationToken);
            return positions.Last();
        }

Client

        private static async Task RunDemo(string gameName)
        {
            var rand = new Random();
            Console.WriteLine("Hit return when the service is up...");
            Console.ReadLine();
            Console.WriteLine("Enter your name:");
            var playerName = Console.ReadLine();

            Console.WriteLine("This might take a few seconds...");
            var gameActor = ActorProxy.Create<IGameActor>(new ActorId(gameName), new Uri(GameActorUri));
            await gameActor.SubscribeAsync<IGameEvents>(new GameEventsHandler(gameActor));

            var playerActorId = await gameActor.JoinGameAsync(playerName, CancellationToken.None);
            var playerActor = ActorProxy.Create<IPlayerActor>(new ActorId(playerActorId), new Uri(PlayerActorUri));

            while (true)
            {
                Console.WriteLine("Press return to move to new location...");
                Console.ReadLine();
                
                await playerActor.MoveToAsync(rand.Next(100), rand.Next(100), CancellationToken.None);
            }
        }

GameEventHandler

        public void ScoreboardUpdated(PlayerInfo lastInfo)
        {
            Console.WriteLine($"Scoreboard updated. (Last move by: {lastInfo.PlayerName})");

            var positions = _gameActor.GetLatestPlayerInfoAsync(CancellationToken.None).ConfigureAwait(false).GetAwaiter().GetResult(); // this hangs forever

            foreach (var playerInfo in positions) // this line never gits hit
            {
                Console.WriteLine(
                    $"Position of {playerInfo.PlayerName} is ({playerInfo.XCoordinate},{playerInfo.YCoordinate})." +
                    $"\nUpdated at {playerInfo.LastUpdate}\n");
            }            
            
        }

But if I wrap the event handler logic inside a Task.Run() it seems to work.

            Task.Run(async () =>
                {
                    var positions = await _gameActor.GetLatestPlayerInfoAsync(CancellationToken.None);

                    foreach (var playerInfo in positions)
                    {
                        Console.WriteLine(
                            $"Position of {playerInfo.PlayerName} is ({playerInfo.XCoordinate},{playerInfo.YCoordinate})." +
                            $"\nUpdated at {playerInfo.LastUpdate}\n");
                    }
                }
            );

Full source code for the demo here https://github.com/dasiths/Service-Fabric-Reliable-Actors-Demo

AFAIK notifications aren't blocking and are not reliable. So I don't understand why my initial implementation doesn't work. The reentrant pattern doesn't apply here as per my understanding either. Can someone explain to me what's going on here? Is it expected behaviour or a bug?