Skip to content

[SYSTEMDS-3054] Modification of broadcast and broadcastSliced#1340

Closed
OlgaOvcharenko wants to merge 34 commits intoapache:masterfrom
OlgaOvcharenko:fed_broadcast3
Closed

[SYSTEMDS-3054] Modification of broadcast and broadcastSliced#1340
OlgaOvcharenko wants to merge 34 commits intoapache:masterfrom
OlgaOvcharenko:fed_broadcast3

Conversation

@OlgaOvcharenko
Copy link
Copy Markdown
Contributor

@OlgaOvcharenko OlgaOvcharenko commented Jul 7, 2021

This PR modifies broadcasts such that after broadcasting data to the worker once (matrix or frame, not scalar) it will be kept on the worker and reused as federated data.
One particular case of broadcastSliced (used for the federated left indexing only, indices to slice are given) is not covered since there is no reason to store the data that will not be of type BROADCAST or PART and is used only once.

@OlgaOvcharenko OlgaOvcharenko changed the title [SYSTEMDS-3054] Modification of broadcast and broadcastSliced [WIP][SYSTEMDS-3054] Modification of broadcast and broadcastSliced Jul 7, 2021
@OlgaOvcharenko OlgaOvcharenko changed the title [WIP][SYSTEMDS-3054] Modification of broadcast and broadcastSliced [SYSTEMDS-3054] Modification of broadcast and broadcastSliced Jul 7, 2021
@OlgaOvcharenko OlgaOvcharenko changed the title [SYSTEMDS-3054] Modification of broadcast and broadcastSliced [WIP][SYSTEMDS-3054] Modification of broadcast and broadcastSliced Jul 12, 2021
@OlgaOvcharenko OlgaOvcharenko changed the title [WIP][SYSTEMDS-3054] Modification of broadcast and broadcastSliced [SYSTEMDS-3054] Modification of broadcast and broadcastSliced Jul 28, 2021
@OlgaOvcharenko
Copy link
Copy Markdown
Contributor Author

Hi @mboehm7 , could you please review the PR and sorry for taking so long. Thanks.

@mboehm7
Copy link
Copy Markdown
Contributor

mboehm7 commented Aug 12, 2021

Thanks @OlgaOvcharenko - I started finalizing this PR and will merge it in tomorrow. As we discussed, we can do the reuse of broadcasts by simple checks inside the original broadcast primitives without changes at federated workers and special handling of broadcast maps.

@OlgaOvcharenko
Copy link
Copy Markdown
Contributor Author

The maps were modified to handle broadcasted data as federated.

@asfgit asfgit closed this in 6649bcd Aug 13, 2021
@mboehm7
Copy link
Copy Markdown
Contributor

mboehm7 commented Aug 13, 2021

Thanks for starting on this @OlgaOvcharenk, I now finalized the PR by removing the remaining broadcast maps and sets at the coordinator and federated workers, removed the broadcast cleanup per instruction, and now these broadcasts are pure federated data and thus can be reused, and ordinary rmvar instructions handle the proper cleanup once a variable leaves its live scope.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants