[SYSTEMDS-2604] Federated compile and federated output propagation

[sebwrede]

This PR adds compilation of federated instructions when OptimizerUtils.FEDERATED_COMPILATION is set to true.
Additionally, a federated output flag is added and propagated through HOPs, LOPs, and instructions. The federated output flag will later be used to change the processing of federated instructions so that the output of federated instructions are not retrieved before all subsequent federated instructions have been executed.
The federated compilation and propagation of the federated output flag will be extended to other operations in future PRs.

Several design decisions have been made in this PR, for instance the decision to add the federated output flag as an additional operand in the instruction string. It could also have been added as a property of the output operand, but this would have consequences on how CPOperands are handled in the system. Comments on this or new ideas are appreciated.

[Baunsgaard]

did you fix some bugs since it parses all tests?

[sebwrede]

did you fix some bugs since it parses all tests?

I removed an empty file, which made one test fail since it did not have an Apache license.
I also ran the failing tests locally, where I discovered that I needed to install some R packages and some Python packages. One of the R packages, imputeTS, failed to install when I ran our R install script, so I had to look up why this happened. I found a solution where I had to run the following:

sudo apt-get install aptitude
sudo apt-get install libcurl4-openssl-dev
sudo apt-get install libxml2-dev

This idea came from here: sagemath/cloud#114 (comment).
After installing the above, I ran the R install again and then the tests passed locally. I then committed and pushed. The only update in that commit was the removal of the empty file.

[Baunsgaard]

did you fix some bugs since it parses all tests?

I removed an empty file, which made one test fail since it did not have an Apache license.
I also ran the failing tests locally, where I discovered that I needed to install some R packages and some Python packages. One of the R packages, imputeTS, failed to install when I ran our R install script, so I had to look up why this happened. I found a solution where I had to run the following:

sudo apt-get install aptitude
sudo apt-get install libcurl4-openssl-dev
sudo apt-get install libxml2-dev

This idea came from here: sagemath/cloud#114 (comment).
After installing the above, I ran the R install again and then the tests passed locally. I then committed and pushed. The only update in that commit was the removal of the empty file.

okay, i will try to reinstall my r because i have tests that fail in GitHub actions and not locally...

[Baunsgaard]

LGTM, only minor syntax comments

[phaniarnab]

LGTM.
For my understanding, did you mean that this flag will help to execute a chain of federated operations before fetching the outputs to the coordinator? Is it not the case today?

[sebwrede]

For my understanding, did you mean that this flag will help to execute a chain of federated operations before fetching the outputs to the coordinator? Is it not the case today?

This PR adds a flag to compile the federated instructions during compile-time instead of simply converting the CP instructions to FED instructions during runtime. Later, the decision to compile to FED will be expanded so that it can base the decision on several scenarios, including the scenario where a chain of federated operations can be executed without sharing the data with the coordinator. Many of the FED instructions are currently implemented in a way that keeps the output federated, but this decision of keeping the output federated needs to be dependent on the flag so that we can have some kind of "FED planner" which compiles this information.
At the moment, the federated instructions do not take the privacy constraints into account. When we have a FED planner, it could discover a privacy constraint on some input data and then compile the instructions accordingly. This may in some cases mean that it does not just blindly pick the chain of federated instructions, but instead considers the privacy constraint and makes a new plan that adheres to the constraint. This is just an example of how the FED planner will be convenient later on.
I implemented this as a federated output flag to begin with, but this will likely be expanded later on since there are more decisions than just federated vs. local. I will deal with this in later PRs when I have a basic FED planner and support federated compilation of more operations.

[phaniarnab]

For my understanding, did you mean that this flag will help to execute a chain of federated operations before fetching the outputs to the coordinator? Is it not the case today?

This PR adds a flag to compile the federated instructions during compile-time instead of simply converting the CP instructions to FED instructions during runtime. Later, the decision to compile to FED will be expanded so that it can base the decision on several scenarios, including the scenario where a chain of federated operations can be executed without sharing the data with the coordinator. Many of the FED instructions are currently implemented in a way that keeps the output federated, but this decision of keeping the output federated needs to be dependent on the flag so that we can have some kind of "FED planner" which compiles this information.
At the moment, the federated instructions do not take the privacy constraints into account. When we have a FED planner, it could discover a privacy constraint on some input data and then compile the instructions accordingly. This may in some cases mean that it does not just blindly pick the chain of federated instructions, but instead considers the privacy constraint and makes a new plan that adheres to the constraint. This is just an example of how the FED planner will be convenient later on.
I implemented this as a federated output flag to begin with, but this will likely be expanded later on since there are more decisions than just federated vs. local. I will deal with this in later PRs when I have a basic FED planner and support federated compilation of more operations.

Got it. Thanks for explaining in detail. Pushing the decision to the compiler seems to be a good idea. Regarding the chain of OPs, many today need partial output aggregation though I guess.

[sebwrede]

Regarding the chain of OPs, many today need partial output aggregation though I guess.

Yes, this is true. There will be a lot of work with implementing the processing of the federated instructions.