[SYSTEMDS-3054] Modification of broadcast and broadcastSliced#1340
[SYSTEMDS-3054] Modification of broadcast and broadcastSliced#1340OlgaOvcharenko wants to merge 34 commits intoapache:masterfrom
Conversation
…nto fed_broadcast3
|
Hi @mboehm7 , could you please review the PR and sorry for taking so long. Thanks. |
|
Thanks @OlgaOvcharenko - I started finalizing this PR and will merge it in tomorrow. As we discussed, we can do the reuse of broadcasts by simple checks inside the original broadcast primitives without changes at federated workers and special handling of broadcast maps. |
|
The maps were modified to handle broadcasted data as federated. |
|
Thanks for starting on this @OlgaOvcharenk, I now finalized the PR by removing the remaining broadcast maps and sets at the coordinator and federated workers, removed the broadcast cleanup per instruction, and now these broadcasts are pure federated data and thus can be reused, and ordinary rmvar instructions handle the proper cleanup once a variable leaves its live scope. |
This PR modifies broadcasts such that after broadcasting data to the worker once (matrix or frame, not scalar) it will be kept on the worker and reused as federated data.
One particular case of broadcastSliced (used for the federated left indexing only, indices to slice are given) is not covered since there is no reason to store the data that will not be of type BROADCAST or PART and is used only once.