Simplified federated statistics workflow, Streaming federated statistics and code refactoring #2265
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… into some utils class 2. Refactoring the WFController class into BaseWFController class and WFController class. The BaseWFController has majority of the logics and implement ControllerSpec. But not implement Responder, which is needed for Server-side controller. WFController Inherited both BaseWFController and Controller. This allows BaseWFController be able to be used on the client-side controller logics without implementing control_flow() method
2. add late result handling 3. change the API spec to allow get result one at a time
…nary sharable or DXO converted Sharable as FLModel convert to shareable involves DXO, Fed Stats controller did not use DXO for input sharable 2. Fix FLModelUtils paramsTypoe checking, instread of raise Exception when params type is None, it should use FULL type when the params is not None 3. refactfore the code base many of the fed stats calculation is placed to fed_stats_utils.py, both statistics_controller.py and FedStatistics workflow can share 5. make the fed stats workflow streamable job.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR is built on top of this PR: #2250. So it must be add after that PR is merged.
The main changes
1, Refactory Statistics_executor to allow the input to be pure Dictionary sharable or DXO converted Shareable
as FLModel convert to shareable involves DXO embedded inside the Shareable;
Fed Stats controller did not use DXO for input shareable
2. Fix FLModelUtils paramsTypoe checking. instead of raise Exception when params type is None, it should use FULL type when the params is not None
3. Refactforing the codebase. Many of the fed stats calculations are now placed to fed_stats_utils.py, both statistics_controller.py and the new simplified FedStatistics workflow can share the code. Both produced the same result.
A few sentences describing the changes proposed in this pull request.
Types of changes
./runtest.sh.