parallel_map as decorator?

[ajavadia]

What is the expected enhancement?

Is this possible? Right now when making a function you have to create a function_X_parallal and a function_X, where the 1st calls the 2nd using parallel_map. It would be nice if one could just write the 2nd, and decorate it to indicate parallel run.

@nonhermitian @atilag thoughts?

[nonhermitian]

Yes uou can. Although, I do bot understand your need for two functions here

[atilag]

I like that approach, as the parallel_map is very generic and it's a very expressive way to state that the function being decorated will be executed concurrently. Also, we could change the concurrency strategy in a future by only changing the decorator implementation. I'd call the decorator something like: @run_concurrent

[nonhermitian]

It is generically useful, and should probably not be confined tonthe transpile dir

[ajavadia]

@nonhermitian by two functions I mean first creating a function task, then creating another wrapper task_parallel, which calls parallel_map(task, args). I think we can do this just by a for loop and a decorator.

I get lost chasing the following code:

_dags_2_dags():
      parallel_map(_transpile_dags_parallel):
            transpile_dag()

[nonhermitian]

You really only need to define one function func that you would like to be able to run in parallel, and then just call parallel_map on it.

I have also tried to make a decorator for parallel_map, but an issue arises that the function being decorated cannot be pickled: PicklingError: Can't pickle <function func at 0xa20f2ef28>: it's not the same object as __main__.func

[nonhermitian]

This foes not work due to the above pickling issue.