[SYSTEMDS-3185] Federated Lookup Table Statistics#1487
Closed
ywcb00 wants to merge 8 commits into
Closed
Conversation
ywcb00
force-pushed
the
feat/fed/multitenancy/flt/stats
branch
2 times, most recently
from
81a70d3 to
9b4c316
Compare
j143
reviewed
Dec 20, 2021
ywcb00
commented
Dec 20, 2021
| } | ||
|
|
||
|
|
||
| DMLScript.USE_LOCAL_SPARK_CONFIG |= dmlconf.getBooleanValue(DMLConfig.USE_LOCAL_SPARK_CONFIG); |
Contributor
Author
ywcb00
commented
Dec 20, 2021
Comment on lines
-107
to
+110
| //FIXME still runs into blocking | ||
| // @Test | ||
| // public void testSumSharedWorkersSP() { | ||
| // runMultiTenantSharedWorkerTest(OpType.SUM, 3, 9, ExecMode.SPARK); | ||
| // } | ||
| @Test | ||
| public void testSumSharedWorkersSP() { | ||
| runMultiTenantSharedWorkerTest(OpType.SUM, 3, 9, ExecMode.SPARK); | ||
| } |
Contributor
Author
There was a problem hiding this comment.
@mboehm7 I wasn't able to reproduce the error you mentioned. Is the blocking state of this test caused by the not set DMLScript.USE_LOCAL_SPARK_CONFIG flag or is there another problem? 🤔
ywcb00
force-pushed
the
feat/fed/multitenancy/flt/stats
branch
2 times, most recently
from
9cc1fac to
ca675b1
Compare
ywcb00
force-pushed
the
feat/fed/multitenancy/flt/stats
branch
from
ca675b1 to
6e2c816
Compare
… coordinators all addressing the same federated workers (sameWorkers)
create alsCG algorithm tests (SameWorkers and SharedWorkers) add runAll scripts chore(FederatedLookupTable.java): add log trace message to log every call to getECM()
…ripts docs(FederatedLookupTable.java): add class and function headers
…testing feat(FederatedMultiTenantTest): add junit java class to test the multi tenancy within the test system add the respective dml script chore(AutomatedTestBase.java): change the access specifier of CONFIG_DIR from private to protected feat(SystemDS-MultiTenant-config.xml): add a systemds config with the federated initialization timeout set to 30 seconds
…ted lookup table entries and get requests add different counts for the different data types in put requests add a bytes count for put requests of matrices and frames refactor(Statistics.java): divide related stats and their methods into own static classes feat(utils/stats/**.java): create static classes for related statistics refactor(**.java): change all calls to statistics according to the outsourced stats into new static classes
…le names inside the statistics classes (stat context is determined already by the class prefix) refactor(**.java): change all calls to the modified statistics function according to the name changes
feat(ParamServStatistics.java): add reset method to reset all properties chore(Statistics.java): call the new created reset methods chore(SparkStatistics.java): reset all properties inside the reset method chore(SparkExecutionContext.java): rename method call to the actual method inside the SparkStatistics (inconsistency introduced by rebasing) chore(SparkParamServUtils.java): rename metho call to the actual method (w/o "PS") inside the ParamServStatistics (inconsistency introduced by rebase)
…ONFIG to the specified DMLConfig value when setting the global flags chore(.gitignore): remove the artifacts of the multi tenant test shell scripts from gitignore again (rebase fail) fix(Statistics.java): reset the FederatedStatistics with the Statistics fix(SparkStatistics.java): increment the Spark collect count with every Spark collect time accumulation chore(LocalInstruction.java): set the outputBufering flag to true in order to obtain the ByteArrayOutputStream from the runTest() call chore(FederatedMultiTenantTest.java): do not redirect the error stream of the coordinator to its stdout since we are obtaining them separate now
ywcb00
force-pushed
the
feat/fed/multitenancy/flt/stats
branch
from
6e2c816 to
cb6a825
Compare
Contributor
|
Thanks for the refactoring @ywcb00. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi,
this PR (1) adds statistics regarding the federated lookup table to the federated statistics, (2) splits the federated PUT request count into separate counts for the individual Object types (e.g., Matrix, Frame, Scalar, and MatrixCharacteristics) and adds a respective byte count for matrices and frames, and (3) refactors the whole Statistics class and moves related statistics into their own statistics class (Codegen, HOP, Native, ParamServ, ParFor, Spark, Transform).
Thanks for review :)