Serialize `numba.typed.List`

[seanlaw]

What happened:

I am trying to call a numba njit function that requires a numba.typed.List as a parameter. Eventually, this numba.typed.List will be filled with numpy arrays of variable length (think jagged array and hence why we are using a list). Calling this numba function works without dask but it fails when dask is used and I am getting a:

TypeError: ('Could not serialize object of type List.', '[[], [], [], [], [], [], [], [], [], []]')

What you expected to happen:

I (naively) expect to be able to serialize this numba.typed.List and then be able to return the numba.typed.List

Minimal Complete Verifiable Example:

import numpy as np
from dask.distributed import Client, LocalCluster
from numba.typed import List
from numba import njit

@njit()
def _some_func(T, A):
    # Do something with T and A
    # Maybe fill A with something like
    # for i in range(len(T)):
    #     A[i] = T[ : i]
    return A


if __name__ == "__main__":
    dask_cluster = LocalCluster(n_workers=2, threads_per_worker=2)
    T = np.random.rand(10)
    n = T.shape[0]
    A = List([np.empty(0, dtype=np.float64) for _ in range(n)])

    with Client(dask_cluster) as dask_client:
        T_future = dask_client.scatter(T, broadcast=True, hash=False)
        A_future = dask_client.scatter(A, broadcast=True, hash=False)

        future = dask_client.submit(
            _some_func,
            T_future,
            A_future,
        )

        results = dask_client.gather(future)
        dask_client.cancel(T_future)
        dask_client.cancel(A_future)

Anything else we need to know?:

Environment:

• Dask version:

dask                      2022.1.1           pyhd8ed1ab_0    conda-forge
dask-core                 2022.1.1           pyhd8ed1ab_0    conda-forge
distributed               2022.1.1         py39h2804cbe_0    conda-forge
cloudpickle               2.0.0              pyhd8ed1ab_0    conda-forge

• Python version: 3.9.10
• Operating System: Mac OS
• Install method (conda, pip, source): conda

Cluster Dump State: