Federated quantiles and central moment#1091
Closed
OlgaOvcharenko wants to merge 6 commits intoapache:masterfrom
Closed
Federated quantiles and central moment#1091OlgaOvcharenko wants to merge 6 commits intoapache:masterfrom
OlgaOvcharenko wants to merge 6 commits intoapache:masterfrom
Conversation
Updated federated min/max Missing bivar.dml Fixed fed min and max Removed ouput Updated dml Fixed dml Fixed FederationUtils Updated bivar Updated federated min/max Fixed fed min and max Removed ouput Updated dml Fixed dml Fixed bivar Fixed comment in bivar.dml Fed min, max, mean, sum Delete MinMax Test Updated federated min/max Fixed fed min and max Removed ouput Updated dml Fed right index Updated fed mean and sum Fixed test Fed sum, mean, minmax and tests Tests Fixed [MINOR] Multi-threaded parfor result file deletion (latency mitigation) After parfor operations a result-merge implementation merges the partial results from parfor workers into the final result variables. In case of remote parfor, we have #result variables x #parfor tasks files, which - in case of in-memory result merge - are into the driver, aggregated, and finally deleted. In sub-optimal cluster configurations, the delete can have substantial latency (independent of file size). To mitigate this latency we now delete these files in a multi-threaded manner, which showed good performance. Note that we refrain from asynchronous deletion to avoid synchronization in case of parfor loops in surrounding while/for loops (where the same files might be written multiple times). [MINOR] Fix various warnings (UINT handling, generics, static) [SYSTEMDS-2681] Federated Bivariate Statistics SYSTEMDS 2543-2544 Federated Aggregations: - Federated Min + Max col and row aggregation - Federated mean and sum aggregations Closes apache#1040 [SYSTEMDS-2680] New ALS built-in functions (als, alsCG, alsDS) AMLS project SS2020. Closes apache#1070. [SYSTEMDS-2682/3] New Lasso and PPCA built-in functions AMLS project SS2020. Closes apache#1071. [SYSTEMDS-2684] Fix built-in function error handling, fix lasso args This patch adds more robust error handling for the scenario of dml-bodied builtin functions whose scripts are are loaded but the functions are unavailable (e.g., due to typos). Furthermore, this also includes a fix of the lasso builtin function, where last minute renaming of function arguments (for consistency) failed the tests. Move Privacy Post-Processing To Be Before Lineage Cache Fed CM, qsort and qpick Federated quantile and CM
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds federated qsort, qpick and cm instructions.