Missing ChildActorExited messages

[andatt]

Hi Kevin,

I think this might just be me being stupid but I am using the following code:

from thespian.troupe import troupe
from thespian.actors import ActorTypeDispatcher, Actor
from thespian.actors import ActorSystem
import logging

class ActorLogFilter(logging.Filter):
    def filter(self, logrecord):
        return 'actorAddress' in logrecord.__dict__


class NotActorLogFilter(logging.Filter):
    def filter(self, logrecord):
        return 'actorAddress' not in logrecord.__dict__


def log_config(log_file_path_1, log_file_path_2):
    return {
        'version': 1,
        'formatters': {
            'normal': {'format': '%(levelname)-8s %(message)s'},
            'actor': {'format': '%(levelname)-8s %(actorAddress)s => %(message)s'}},
        'filters': {'isActorLog': {'()': ActorLogFilter},
                    'notActorLog': {'()': NotActorLogFilter}},
        'handlers': {'h1': {'class': 'logging.FileHandler',
                            'filename': log_file_path_1,
                            'formatter': 'normal',
                            'filters': ['notActorLog'],
                            'level': logging.INFO},
                     'h2': {'class': 'logging.FileHandler',
                            'filename': log_file_path_2,
                            'formatter': 'actor',
                            'filters': ['isActorLog'],
                            'level': logging.INFO}, },
        'loggers': {'': {'handlers': ['h1', 'h2'], 'level': logging.DEBUG}}
    }


class PrimaryActor(ActorTypeDispatcher):

    def receiveMsg_dict(self, msg, sender):
        if not hasattr(self, "secondary_actor"):
            self.secondary_actor = self.createActor(SecondaryActor)
        else:
            logging.info("sending from primary...")
            self.send(self.secondary_actor, msg)

    def receiveMsg_ChildActorExited(self, message, sender):
        logging.info("A child of actor {} has exited...".format(
            self.__class__.__name__
        ))


@troupe(max_count=20, idle_count=1)
class SecondaryActor(ActorTypeDispatcher):
    received_message_count = 0

    def receiveMessage(self, msg, sender):
        logging.info("received in secondary...")

    def receiveMsg_ChildActorExited(self, message, sender):
        logging.info("A child of actor {} has exited...".format(
            self.__class__.__name__
        ))

thespian_system = ActorSystem(
    "multiprocTCPBase",
    {},
    logDefs=log_config("bug_check_1.log", "bug_check_2.log")
)

primary_actor = thespian_system.createActor(PrimaryActor)


for x in range(0, 1000):
    message = {"number": x}
    thespian_system.tell(primary_actor, message)

I would expect to see some logging messages stating the child actors have exited but I get nothing. I just see the send / received messages. Is there something wrong with the code?

Thanks

[kquick]

The troupe leader handles the ChildActorExited messages for all of the troupe workers, so your ChildActorExited handler in SecondaryActor never gets invoked because those aren't passed through from the troupe leader management. The troupe leader never exits, so PrimaryActor doesn't receive a ChildActorExited.

The troupe workers are ephemeral, being created on demand and destroyed on lack of demand, so each one can create assistant actors but the assistant actors will be destroyed along with themselves. The workers will receive those ChildActorExited messages, so if SecondaryActor had created child actors you would have seen logging for those.

[kquick]

Also I realize this confusion might have been caused by some of my earlier advice on the other issue where I did recommend handling the ChildActorExited, so I apologize for the mixed messaging. With the recent change in the troupe that you inspired, the troupe management is more reliable and should work as I described above, and I have taken an action item to document this more clearly.

[andatt]

Ahh I think I understand. The reason this came up was because some tests broke in a subtle way after the recent upgrade. The tests were relying on picking up log entries triggered by ChildActorExited. I thought it was maybe something I had done unwittingly but it sounds like the Thespian changes make this approach no longer possible in determining when a child actor has finished its work.

I will take a look at the code and see if I can modify the test to find a different way of detecting once the work has been done.

[andatt]

hmm....I think this could be quite difficult to fix without knowing when a child actor has exited. So just to be clear, the troupe leader and the secondary actor are two completely different entities? And the troupe leader has no way of sharing the fact that a troupe member has exited with the secondary actor?

[kquick]

The @troupe decorator causes Thespian to create a troupe leader whenever a createActor() is called for that actor name. The troupe leader has the same name, but uses an internally supplied receiveMessage that bypasses the one you have defined. Anytime that internal receiveMessage gets a message from some other actor, it creates a worker of that actor type and forwards the message to the worker for handling.

The idea is that you can write your SecondaryActor as a normal (non-troupe) actor and things should run just fine. If you discover that SecondaryActor is a bottleneck in your system and it is feasible for it to parallelize the work for messages it receives, you can simply add the @troupe decorator and the parallelization is handled for you by the decorator creating the troupe leader functionality and passing messages along to the workers created on-demand.

One way to clarify your development needs might be to remove/comment-out the @troupe decorator and get things running (albeit more slowly) with a single SecondaryActor. Then once things are working to your satisfaction, you can re-add the @troupe decorator to enable parallel SecondaryActors for more throughput.

The only real considerations when adding the @troupe decorator are:

1. Whether messages cause internal state changes that affect future message processing, and
2. Whether a message is completely processed by the time the receiveMessage method exits.

As an example of #1:

class StatefulActor(Actor):
    def __init__(self, *args, **kw):
        super(self, StatefulActor).__init__(*args, **kw)
        self.msgcount = 0
    def receiveMessage(self, msg, sender):
        self.msgcount = self.msgcount + 1
        self.send(sender, self.msgcount)

If I send 10 messages to the StatefulActor above, I will get responses of: 1, 2, 3, ..., 10
However, if I add a @troupe designator, then the responses will be indeterminate and look something like: 1, 2, 1, 3, 2, 1, 4, 2, 1, 3

The StatefulActor is not a good candidate for the @troupe decorator. There are ways to make additional adjustments to allow a troupe configuration, but these involve more complicated management and update of the associated state.

As an example of #2:

class HelperActor(Actor):
    def receiveMessage(self, msg, sender):
        ...
        self.send(sender, SomeResponseMessage(...))

@troupe
class TwoStageActor(Actor):
    def receiveMessage(self, msg, sender):
        if isinstance(msg, SomeResponseMessage):
            self.send(self.requester, final_response(msg))
            self.troupe_work_in_progress = False   # see below
        else:
            if not getattr(self, 'helper', None):
                self.helper = self.createActor(HelperActor)
            self.requester = sender
            ...
            self.send(self.helper, msg)
            self.troupe_work_in_progress = True   # see below

In this example, the TwoStageActor cannot complete the request until it gets additional information from the HelperActor. To indicate this, it sets the troupe_work_in_progress attribute to true, which keeps the troupe leader from sending it more work until it sets that attribute to false. Adding the @troupe decorator to your SecondaryActor may require judicious updates to troupe_work_in_progress. Note that if you don't have this attribute, it is assumed to be false.

If your use case is more complicated, then the simple @troupe parallelization mechanism may not be appropriate, and it may be better to use your own manager to coordinate activities as appropriate to your workflow.

[andatt]

Thanks for your detailed response on that Kevin. I forgot I hadn't come back and closed this one. I solved the issue so thanks!