Support Python multiprocessing within assets

[PeterJCLaw]

What's the use case?

It would be great if Python's multiprocessing (and thus things like concurrent.futures etc.) were supported within asset functions. Currently I get a ModuleNotFoundError relating to the name of my code location when the new process tries to un-pickle its data.

Ideas of implementation

I'm guessing the issue is to do with how the code locations are added to sys.path, or the lack of doing that? Manually ensuring the code location is on sys.path seems to make things work. The following is a liberal (untested) cut-down based on changes I've made locally which got this working for my use-case. The not-great bit is that I'm hard-coding the code-location path, which I'd like to avoid.

import sys
import time
from concurrent.futures import ProcessPoolExecutor

import tqdm
import dagster


def f(x: int) -> int:
    time.sleep(0.1)
    return x * 2


@dagster.asset()
def demo() -> None:
    sys.path.append('...')

    with ProcessPoolExecutor(max_workers=3) as pool:
        N = 23
        print(list(tqdm.tqdm(
            pool.map(f, range(N)),
            total=N,
        )))

Additional information

I'm running the materialization via the CLI, something like:

$ dagster asset materialize --working-directory ./my-dagster/code-location/ -m code_location --select assets/demo

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[joaorulff]

Any updates on this?