task mapping on generators?

[mohamed--abdel-maksoud]

Is there a reason why we can't pass a generator expression to task.map? loading the whole list in memory is potentially inefficient.
If what is needed is just development time, I'd like to work on it.

[joshmeek]

Hey @mohamed--abdel-maksoud the core development team has a plan in the works for a larger mapping refactor to better support some use cases we have seen arise as well as enable the addition of new features. This may end up solving your use case but I can't say for certain. That being said the reason why mapping over generators is not currently supported is due to the nature of Prefect requiring tasks to finish and return pickleable results for their downstream tasks.

[mohamed--abdel-maksoud]

Thanks @joshmeek for the quick answer, I'll wait for the refactoring then.
I understand a task cannot return a generator since it's not pickle'able, in my case though the generator is returned by a vanilla function.

[earthastronaut]

@joshmeek I also ran into this. I'm interested to see how you solve it. My use case: I have a task which preforms an unbounded task (looping over request pages) and generates results which a downstream task will process without blocking.

Looking at the code; It's fairly straight forward to modify prefect.engine.task_runner.TaskRunner.run_mapped_task to handle next(upstream_state.result) generator with StopIteration in addition to the upstream_state.result[i]. However, it doesn't start the downstream task until the upstream finishes as you pointed out. And the if the upstream generator is truely unbounded it may never end which could be a problem. It could queue up new downstream tasks by executor.submit but that likely breaks some of the benefits of requiring upstream processes to finish (like check points) which honestly I don't fully understand yet.

Anyway, I hope that adds something to the conversation. Thanks for working on this hard problem and making my development easier!

[michcio1234]

@joshmeek is there any timeline for this?

[cicdw]

Chiming in for josh here -- there is currently no timeline for this although it is something we are increasingly interested in undertaking! The implementation will most likely be more general than mapping over generators, and allow for a "parent" task to asynchronously spawn new tasks that can immediately begin running (under the right executor). I will update here if and when this enters the planning phases.

[sveinugu]

@joshmeek Seems this issue was closed by mistake (commit references pull request 2023, not issue 2023). Any general support for generators still on the table?

[zanieb]

@sveinugu that commit is indeed unrelated but this was closed long ago by the opener of the issue. I believe mapping over generators is supported in Prefect v2.

[sveinugu]

@sveinugu that commit is indeed unrelated but this was closed long ago by the opener of the issue. I believe mapping over generators is supported in Prefect v2.

Thanks for the feedback. I see I misread the issue a bit. #5708 is what I was looking for.