Allow cross process communication using custom messages

[nathan-beam]

This pull fixes #1780 and #1506.

Added two methods, register_message and send_message to the two DistributedRunner classes.

register_message takes a custom message type and a callback function to call when the specified message is received. The callback's parameters are environment (obviously the runner's environment) and msg, which is the message itself.

send_message takes a custom message type and optional data to send with the message, allowing a user to easily send their own custom messages.

Updated tests/documentation where necessary, added an example implementation to the examples directory (examples\custom_messages.py).

[cyberw]

Cool stuff. You saw the linting/formatting issues right? I'd like to hold off on this until we have merged and released the new worker distribution style though. So 2.1 at the earliest.

[nathan-beam]

Any ETA on 2.1? I have some functionality that would depend on this, so just want to have a timeline so I can plan things out as best I can. Thanks!

[cyberw]

It mostly depends on that other PR, and I’m not writing that myself. I’d say maybe a couple of weeks, maybe more.

[cyberw]

A small note: please write commit messages in a way that makes sense even if the reader doesnt know what branch/PR they were part of.

[cyberw]

Maybe some test cases for when a message is sent but no receiving method is registered? (should ideally log a warning)

[pappacena]

+1 for this PR.

I'm facing a similar situation, and I had to mostly copy the WorkerRunner.worker method in a subclass.

I was preparing a refactoring to make it easier to subclass WorkerRunner and add new message types, but this proposal is way better!

Good job!

[cyberw]

@nathan-beam If you add the test case I mentioned I will merge this and it will become part of the next minor version. No point in waiting for that other PR (and this is not a breaking change anyway).

[nathan-beam]

@cyberw Added. Let me know if you think it's missing anything else

[cyberw]

Looks pretty good now.

Just one thing: as most plans are testes in a stand alone environment before moving to distributed, could this be made to work on stand alone (LocalRunner) runs?

[nathan-beam]

Just one thing: as most plans are testes in a stand alone environment before moving to distributed, could this be made to work on stand alone (LocalRunner) runs?
@cyberw

Not sure what "working" would look like on a local runner, since there isn't any messaging infrastructure. Are you alright with just logging a warning? The alternative I can think of is to covertly register a new event and have the send_message() method fire that event to emulate sending/receiving a message.

[cyberw]

It would be great if we could emulate sending/receiving, so that the same test plan can work standalone and distributed without changes. I havent thought about it too much though, but see if you can't solve it without resorting to something that feels too hacky (if it is impossible/very hard then lets not do it)

[nathan-beam]

Was actually even easier than I was thinking. Just moved the custom messages dictionary and registration the base Runner class and simply called the registered callback from the LocalRunner's send_message() function. Should be 100% functional between Master/Worker and Local modes now 👍

[cyberw]

Nice work!