Skip to content

[SYSTEMDS-3374] Federated primitive for transferring a local data object to a federated representation#1609

Closed
OlgaOvcharenko wants to merge 10 commits into
apache:mainfrom
OlgaOvcharenko:transfer_local
Closed

[SYSTEMDS-3374] Federated primitive for transferring a local data object to a federated representation#1609
OlgaOvcharenko wants to merge 10 commits into
apache:mainfrom
OlgaOvcharenko:transfer_local

Conversation

@OlgaOvcharenko
Copy link
Copy Markdown
Contributor

@OlgaOvcharenko OlgaOvcharenko commented May 9, 2022

This PR adds a federated primitive for transferring a local data object (matrices and frames) to a federated representation, e.g. federated(local_matrix=A1, addresses=list($in_X1, $in_X2, $in_X3, $in_X4), ranges=list(list(0, 0), list($rows/4, $cols), list($rows/4, 0), list(2*$rows/4, $cols), list(2*$rows/4, 0), list(3*$rows/4, $cols), list(3*$rows/4, 0), list($rows, $cols))).

@OlgaOvcharenko OlgaOvcharenko changed the title [WIP][SYSTEMDS-3374] Federated primitive for transferring a local data object to a federated representation [SYSTEMDS-3374] Federated primitive for transferring a local data object to a federated representation May 26, 2022
@OlgaOvcharenko OlgaOvcharenko marked this pull request as ready for review May 26, 2022 00:27
@mboehm7
Copy link
Copy Markdown
Contributor

mboehm7 commented Jun 5, 2022

LGTM - thanks for the patch @OlgaOvcharenko. I will merge this in with minor changes as a number of merge conflicts already accumulated. However, there is quite some redundancy in InitFEDInstruction - could you please (in a follow-up PR) eliminate this redundancy, construct a federatedMap, and subsequently simply call broadcastSliced as an existing primitive for transferring slices of a local data object to individual federated workers. That way, we're reuse the well-tested and compact runtime primitive without need for changing the worker handler.

@asfgit asfgit closed this in 641949d Jun 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants