Add prototype support for an ak.apply function

[jabraham17]

Adds prototypical support for calling an arbitrary python function on a distributed Arkouda array.

For example:

import arkouda as ak
ak.connect()
arr = ak.array([1,2,3])
res = ak.apply(arr, lambda x: x+1)
# res is now [2, 3, 4]

This PR uses cloudpickle to pickle arbitrary python code. The pickle is shipped to the server as base64, where it is depickled and executed.

To use this module, there are new requirements for Arkouda.

• The arkouda_server must be built with the same version of python that it will be run with.
• The arkouda_server's python version and the client's python version must be the same major/minor version.
• cloudpickle must be available to the arkouda_server, not just the client
• Any dependencies used within the body of the applied function must be available to the arkouda_server

[jabraham17]

The CI failures appear unrelated to the contents of this PR

[e-kayrakli]

@jabraham17 -- thanks for this PR. I left some comments.
@ajpotts -- I see that Jade has some questions for you in the PR, could you take a look?

[e-kayrakli]

Some libraries seem to not play with being cloudpickled multiple times. For example, using numpy inside a function applied to a pdarray is fine. However, if running from a jupter notebook and the apply is run from a cell twice in a row, the server will segfault

Do you have an idea how to mitigate this? It sounds like the second time you cloudpickle a numpy function, it is not playing well with the server, is that correct? Could the server cache functions? Arguably, that will not prevent the client from resending them, but they can be ignored. Or ideally, client can maintain a state, in which the fact that a given function is cached by the server can be stored.

[jabraham17]

Do you have an idea how to mitigate this? It sounds like the second time you cloudpickle a numpy function, it is not playing well with the server, is that correct? Could the server cache functions? Arguably, that will not prevent the client from resending them, but they can be ignored. Or ideally, client can maintain a state, in which the fact that a given function is cached by the server can be stored.

Its not just the use of the same function, its about the use of a module in a function. This snippet replicates the issue

import arkouda as ak
import numpy as np

ak.connect()

n = 100_000
arr = ak.randint(0, 10, n)

def foo(x):
    np
    return 2.0
res = ak.apply(arr, foo, "float64")
print(res)

def bar(x):
    np
    return 1.0
res = ak.apply(arr, bar, "float64")
print(res)

[jabraham17]

I have tracked down the crash to an issue with numpy and multiple interpreters: numpy/numpy#28271. This error should not crash the server (only throw an exception), so there is something on the Chapel Python interop side to fix. But this will just make a nicer error message, it will not fix the problem. For that, we either need to reusue the interpreter for each locale or numpy needs to be fixed.

[jabraham17]

This error should not crash the server (only throw an exception), so there is something on the Chapel Python interop side to fix

This is a general Chapel bug, there is nothing that can be done within the Python module to fix it

[ajpotts]

I love this so much!

Unfortunately I don't think we can merge it in until we can reduce the potential for server crashes.

[ajpotts]

Looks good! Thanks for all the hard work on this one.