BroadcastView map Not Implemented

[dalbabur]

The documentation made it seem like BroadcastView.map had been implemented. Is this feature coming?

[minrk]

I don't think anyone is working on it at the moment, but it should be possible.

[dalbabur]

What would it take to implement it? I could attempt to with some directions.

[minrk]

map needs to partition inputs, so that each engine gets its chunk of the input sequence(s), which is done here. In DirectView, this is implemented by partitioning the input in the client and then sending a different message to each endpoint.

In BroadcastView, the client sends just one message (that's kind of its whole deal), so the partitioning would need to be handled in the BroadcastScheduler, instead. Reconstruction could either happen in the scheduler, or only aggregated in the scheduler and reconstructed finally in the client.

Perhaps the simplest implementation would be to still compute the partition in the client, but then send all partitions to the scheduler. Then, as each scheduler relays its messages, it selects the subset of chunks it needs to send along each path, and returns the results in a similar shape for the client to reconstruct.

All that said, you can use BroadcastView to run maps today, as long as you do the partitioning separately, as either SPMD-style partitioning on engines (works best for BroadcastView), or via e.g. DirectView.scatter.

here's a notebook that implements map by combining BroadcastView and DirectView. We could actually ship a simplest-possible BroadcastView.map that is exactly the broadcast_map in that notebook (with some more unique temp variable names to avoid collisions). It's not as efficient as a 'true' Broadcast Map, but it does work, and can always be optimized later.