Akka.Remote: can't reply back remotely to child of Pool router

[pcwiek]

Let's say I have an actor selection inside another actor:

 // populated when ClusterEvent.MemberUp message is received
 var memberAddress = msg.Member.Address;
 var permissions = Context.System.ActorSelection(memberAddress + "/user/permissions");

And then I tell it to do something at one point:

 permissions.Tell(new GetSomePermissions("Abc"));

That actor is hosted inside a local router, so the actor issuing the Tell will have a path of, for example, akka.tcp://my-system@127.0.0.1:22000/user/general/$b.

For the record, I'm using Akka.Cluster.

In the permissions actor, I save the original sender, do some processing by bouncing messages back and forth, and at one point I tell the reply to the original sender.

IActorRef tell

    Receive<SomePermissionStuff>(msg => 
    {
        originalSender.Tell(new Permissions("Abc"));    // doesn't work, originalSender is IActorRef
    });

First tell:

• On the 'General' actor side (tell):

  2015-04-21 12:23:36.274 -04:00 [Debug] "RemoteMessage: ActorSelectionMessage - Message: Shared.General.Permissions.GetSomePermissions - WildCartFanOut: False - Elements: user/permissions to [[akka.tcp://my-system@127.0.0.1:21300/]]<+[akka.tcp://my-system@127.0.0.1:21300/] from [[akka://my-system/user/general/projects/$c]]"
  

• On the 'Permissions' actor side (receive):

  2015-04-21 12:23:36.403 -04:00 [Debug] received local message ["RemoteMessage: ActorSelectionMessage - Message: Shared.General.Permissions.GetSomePermissions - WildCartFanOut: False - Elements: user/permissions to [akka://my-system/]<+akka://my-system/ from [akka.tcp://my-system@127.0.0.1:21400/user/general/projects/$c]"]
  

So far, so good.

'Reply' tell:

• On the 'Permissions' actor side (tell):

  2015-04-21 12:23:38.119 -04:00 [Debug] "RemoteMessage: Shared.General.Permissions.SomePermissionStuff to [[akka.tcp://my-system@127.0.0.1:21400/user/general/projects/$c]]<+[akka.tcp://my-system@127.0.0.1:21400/user/general/projects/$c] from [[akka://my-system/user/permissions]]"
  

• On the 'General' actor side (receive)

  2015-04-21 12:23:38.151 -04:00 [Debug] received local message ["RemoteMessage: Shared.General.Permissions.SomePermissionStuff to [akka://all-systems/]<+akka://all-systems/ from [akka.tcp://my-system@127.0.0.1:21300/user/permissions]"]
  

As you can see, that message goes straight to all-systems - that's not right...

When I change the final reply to...

ActorSelection tell

Receive<SomePermissionStuff>(msg => 
{
    var selection = Context.System.ActorSelection(originalSender.Path);
    selection.Tell(new Permissions("Abc")); 
});

the first tell is exactly the same, but the 'reply' tell changes to:

• On the 'Permissions' actor side (tell)

  2015-04-21 12:23:38.145 -04:00 [Debug] "RemoteMessage: ActorSelectionMessage - Message: Shared.General.Permissions.SomePermissionStuff - WildCartFanOut: False - Elements: user/general/projects/$c to [[akka.tcp://my-system@127.0.0.1:21400/]]<+[akka.tcp://my-system@127.0.0.1:21400/] from [[akka://my-system/user/permissions]]"
  

• On the 'General' actor side (receive)

  2015-04-21 12:23:38.804 -04:00 [Debug] received local message ["RemoteMessage: ActorSelectionMessage - Message: Shared.General.Permissions.SomePermissionStuff - WildCartFanOut: False - Elements: user/general/projects/$c to [akka://my-system/]<+akka://my-system/ from [akka.tcp://my-system@127.0.0.1:21300/user/permissions]"]
  

Now that works without a hitch.

IActorRef path seems to include all elements directly in the path, while ActorSelection has elements defined separately, but in the end they both end up being exactly the same after assembling the full path...

[Aaronontheweb]

If what @pcwiek reported is true, this is a severe bug. Can anyone else replicate?

[pcwiek]

For the record, I'm using Akka.Cluster. I added that information to the original issue.

It's possible that I messed something up, but from 'the outside' - even from the debugger - everything looks correct on the 'Permissions' actor side, paths and everything, but the last reply message just doesn't appear on the 'General' actor side. Needless to say, I was a bit confused. :)

[Aaronontheweb]

@pcwiek sounds like it might be a serialization issue with IActorRef not being reconstituted on the other end of the wire correctly (cc @rogeralsing)

Where does originalSender get populated? Any reason why you're not just using Sender.Tell ?

[pcwiek]

@Aaronontheweb On the permissions side, here's how it looks like (brief version)

public class PermissionsActor : ReceiveActor, IWithUnboundedStash
{
    private IActorRef originalSender;
    private GetSomePermissions req;

    (... snip ...)

    public IStash Stash {get; set;}

    public PermissionsActor()
    {
        Ready();
    }

    private void Ready()
    {
        Receive<GetSomePermissions>(msg => 
        {
            req = msg;
            originalSender = Sender; // <--- here, original sender just captures the... well, original sender
            Become(Busy);
            // push processing to a child actor, setup cut out for brevity
            childActor.Tell(new CheckSomeOtherStuff(msg.Name));
        });
    }

    private void Busy()
    { 
        Receive<GetSomePermissions>(msg => Stash.Stash());

        // This message should come from the child actor's processing task
        Receive<IntermediateResult>(msg => 
        {
            // when I hover over original sender, the debugger shows RemoteActorRef as implementation..?
            // anyway, message below should go back to 'original' sender as a result.
            originalSender.Tell(new SomePermissionStuff(req.Name, msg.Permissions));
            BecomeReady();
        });
    }

    private void BecomeReady()
    {
        originalSender = ActorRefs.NoSender;
        req = null;
        Become(Ready);
        Stash.UnstashAll();
    }
}

[rogeralsing]

    Receive<SomePermissionStuff>(msg => 
    {
        originalSender.Tell(new Permissions("Abc"));    // doesn't work, originalSender is IActorRef
    });

Why doesn't that work?
Are you missing a using on Akka.Actor for the extension method of Tell that use the implicit sender?

Or do you mean "doesnt work" as in it buggs out?

[pcwiek]

@rogeralsing I mean "doesn't work" as in it doesn't deliver the message. The actor doesn't crash or anything, the message is sent:

2015-04-21 12:23:38.119 -04:00 [Debug] "RemoteMessage: Shared.General.Permissions.SomePermissionStuff to [[akka.tcp://my-system@127.0.0.1:21400/user/general/projects/$c]]<+[akka.tcp://my-system@127.0.0.1:21400/user/general/projects/$c] from [[akka://my-system/user/permissions]]"

but on the other side it goes to all-systems instead of my-system/user/general/projects/$c:

2015-04-21 12:23:38.151 -04:00 [Debug] received local message ["RemoteMessage: Shared.General.Permissions.SomePermissionStuff to [akka://all-systems/]<+akka://all-systems/ from [akka.tcp://my-system@127.0.0.1:21300/user/permissions]"]

Unless I use ActorSelection - everything works fine then.

Now I noticed that when changing the names I might have made a clerical error in message type names - anyway, the right message type is sent and handled, because ActorSelection created from the exact same actor path works just fine and delivers the message.

[Aaronontheweb]

Hmmm... if this is true, then https://github.com/akkadotnet/akka.net/blob/dev/src/core/Akka.Remote.Tests/RemotingSpec.cs#L150 should fail.

I'm going to try running that test fixture with Akka.Cluster enabled and see what happens....

[Aaronontheweb]

Ran this spec in Akka.Cluster - had no issues. Could reply back to ActorRef with no problems.

[pcwiek]

@Aaronontheweb I dug deeper and it seems I found something interesting.

If the nested/child actor is created 'as is' by its parent, IActorRef.Tell reply message seems to be handled correctly:

// in 'General' actor
var projectsChild = Context.Child("projects");

// the 'projects' actor will be issuing the remote Tell first to check for permissions
projectsActor = projectsChild .Equals(ActorRefs.Nobody)
            // I'm using DI.
            // 'resolver' is IDependencyResolver passed via constructor
            ? Context.ActorOf(resolver.Create<ProjectsActor>(), "projects")
            : projectsChild;

But, as soon as I change that to include a router:

var projectsChild = Context.Child("projects");
projectsActor = projectsChild .Equals(ActorRefs.Nobody)
            ? Context.ActorOf(resolver.Create<ProjectsActor>().WithRouter(new RoundRobinPool(10)), "projects")
            : projectsChild;

the reply to that actor (well, to one of the actors from the pool specifically, which is visible in the path in original post as general/projects/$c) seems to go nowhere - that is exactly the situation I initially presented.

Having a 'local' router for project retrieval is pretty much must-have, because I don't want to queue all users' project requests (which also involves a remote permissions check with a small state-machine [Ready/Busy] first and subsequent DB access using child actor) in one actor... :)

Also, the whole processing is initiated by web app issuing an Ask to a local proxy actor that just forwards it to remote actors. Could that be the problem?

[Aaronontheweb]

@pcwiek interesting - I wonder if the issue might be that routers don't do a good job resolving the path back from a RemoteActorRef. I know user-defined child actors don't have this problem.

Let me write a test for this real quick and get back to you ;)

[Aaronontheweb]

Can confirm - this is a bug. Reproduced in #892. Working on investigating now.

[Aaronontheweb]

Going to leave this as "up for grabs" in case another contributor wants a stab at it

[Aaronontheweb]

@pcwiek thanks for reporting this! We'll ship Akka.NET v1.0.1 with the fix included.