Calling custom Python code from .NET

To call custom Python code, first copy the file PythonWorker.py and rename it to some meaningful name.

Then, change the following lines:

id = "PythonEcho"
group = ""
max_concurrency = 1
timeout = 60.0

def Computation(input):
    return int(input) * 2

The Computation function processes input and returns output. All Python processing logic goes there.

After that, in .NET create a new class that inherits from PythonActor:

public class PythonEchoActor : PythonActor {

        public PythonEchoActor() : base("PythonWorker.py", id: "PythonEcho", maxConcurrencyPerCpu: 1) {
        }
    }

The id parameters must match between the Python file and PythonActor base constructor. The first constructor parameter is the full path to the Python file. The maxConcurrencyPerCpu parameter tells how many Python processes should be launched per a logical CPU (when zero, only one process is launched).

Now, everything is ready. Create an instance of PythonEchoActor and start it:

var actor = new PythonEchoActor();
actor.Start();

To send a piece of work, use the async PostAndGetResult method. The unit test below shows concurrent execution of 20k simple tasks sent to Python workers.

[Test]
public async void CouldReceiveEchoFromPython() {
    var actor = new PythonEchoActor();
    actor.Start();
    var sw = new Stopwatch();
    sw.Start();
    List<Task<string>> tasks = new List<Task<string>>();
    for (int i = 0; i < 20000; i++) {
        tasks.Add(actor.PostAndGetResult(i.ToString()));
    }
    await Task.WhenAll(tasks);
    sw.Stop();
    for (int i = 0; i < 20000; i++) {
        Assert.AreEqual((i * 2).ToString(), tasks[i].Result);
    }
    Console.WriteLine($"Elapsed: {sw.ElapsedMilliseconds}");
}

On my 4-core machine the test runs in c.5 seconds, or at 4k messages per second.

To use it on different machines, call actor.Start(); only on the worker machines. The method PostAndGetResult works without starting an actor - it will queue a job in Redis and will await its completion for a specific timeout.