Skip to content

Add simple progress log #29

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
come2ry merged 10 commits into from
Sep 11, 2022
Merged

Add simple progress log #29

come2ry merged 10 commits into from
Sep 11, 2022

Conversation

@arukuka
Copy link
Member

@arukuka arukuka commented Sep 6, 2022 •
edited
Loading

Summary

Purpose

Contents

Testing Methods Performed

hjoin

  • make test t=./LibClient/ p=benchmark 
    • diff --git a/Test/LibClient/src/benchmark/test_statistics.py b/Test/LibClient/src/benchmark/test_statistics.py
index 8d90801..3a4f197 100644
--- a/Test/LibClient/src/benchmark/test_statistics.py
+++ b/Test/LibClient/src/benchmark/test_statistics.py
@@ -29,6 +29,7 @@ bench_size = [
     (5),
     (50),
     (500),
+    (1000),
 ]
 
 
diff --git a/Test/docker-compose.yml b/Test/docker-compose.yml
index 4e15f91..ff58978 100644
--- a/Test/docker-compose.yml
+++ b/Test/docker-compose.yml
@@ -1242,7 +1242,7 @@ services:
       - type: bind
         source: ./LibClient/src
         target: /LibClient/src
-    command: ["/bin/bash", "-c", "pipenv run pytest src/benchmark -s -v -log-cli-level=DEBUG"]
+    command: ["/bin/bash", "-c", "pipenv run pytest src/benchmark -s -v -log-cli-level=DEBUG -k hjoin"]
     network_mode: "host"
     depends_on:
       dev_mc1:
  • log (N=1000) 
    computation_container1  | [2022-09-09 09:30:05.347] [info] method id is: 9
computation_container1  | [2022-09-09 09:30:05.513] [info] [progress] hjoin: core (0/1)
computation_container1  | [2022-09-09 09:30:10.552] [info] [progress] hjoin: core (0/1):  7.00 %
computation_container1  | [2022-09-09 09:30:15.648] [info] [progress] hjoin: core (0/1): 16.00 %
computation_container1  | [2022-09-09 09:30:20.853] [info] [progress] hjoin: core (0/1): 25.00 %
computation_container1  | [2022-09-09 09:30:26.188] [info] [progress] hjoin: core (0/1): 34.00 %
computation_container1  | [2022-09-09 09:30:31.387] [info] [progress] hjoin: core (0/1): 43.00 %
computation_container1  | [2022-09-09 09:30:36.481] [info] [progress] hjoin: core (0/1): 52.00 %
computation_container1  | [2022-09-09 09:30:41.650] [info] [progress] hjoin: core (0/1): 61.00 %
computation_container1  | [2022-09-09 09:30:46.785] [info] [progress] hjoin: core (0/1): 70.00 %
computation_container1  | [2022-09-09 09:30:51.865] [info] [progress] hjoin: core (0/1): 79.00 %
computation_container1  | [2022-09-09 09:30:57.039] [info] [progress] hjoin: core (0/1): 88.00 %
computation_container1  | [2022-09-09 09:31:02.175] [info] [progress] hjoin: core (0/1): 97.00 %
computation_container1  | [2022-09-09 09:31:03.795] [info] [progress] hjoin: core (1/1)
computation_container1  | [2022-09-09 09:31:04.493] [info] end_job_id is 4

linear regression

  • use internal 100k sample data
  • log 
    computation_container1  | [2022-09-09 08:43:26.893] [info] method id is: 5
computation_container1  | [2022-09-09 08:43:31.385] [info] [progress] Linear Regression: learning: pre-processing (0/4)
computation_container1  | [2022-09-09 08:43:31.438] [info] [progress] Linear Regression: learning: mat mul (1/4)
computation_container1  | [2022-09-09 08:43:31.505] [info] [Cc2Bts]        Client http://beaver_triple_service:50020 is Active
computation_container1  | [2022-09-09 08:43:58.778] [info] [progress] Linear Regression: learning: mat inverse (2/4)
computation_container1  | [2022-09-09 08:43:59.019] [info] [progress] Linear Regression: learning: mat mul (3/4)
computation_container1  | [2022-09-09 08:44:36.353] [info] [progress] Linear Regression: learning: post-procesing (4/4)
computation_container1  | [2022-09-09 08:44:36.659] [info] end_job_id is 1
computation_container1  | [2022-09-09 08:44:37.871] [info] [Predict]                      Received
computation_container1  | [2022-09-09 08:44:37.871] [info] Model Manager: Model Id is 1
computation_container1  | [2022-09-09 08:44:42.256] [info] [progress] Linear Regression: predict: pre-processing (0/2)
computation_container1  | [2022-09-09 08:44:42.283] [info] [progress] Linear Regression: predict: compute (1/2)
computation_container1  | [2022-09-09 08:44:47.514] [info] [progress] Linear Regression: predict: compute (1/2): 11.00 %
computation_container1  | [2022-09-09 08:44:52.695] [info] [progress] Linear Regression: predict: compute (1/2): 24.00 %
computation_container1  | [2022-09-09 08:44:57.825] [info] [progress] Linear Regression: predict: compute (1/2): 36.00 %
computation_container1  | [2022-09-09 08:45:03.073] [info] [progress] Linear Regression: predict: compute (1/2): 49.00 %
computation_container1  | [2022-09-09 08:45:08.177] [info] [progress] Linear Regression: predict: compute (1/2): 62.00 %
computation_container1  | [2022-09-09 08:45:13.365] [info] [progress] Linear Regression: predict: compute (1/2): 74.00 %
computation_container1  | [2022-09-09 08:45:18.626] [info] [progress] Linear Regression: predict: compute (1/2): 87.00 %
computation_container1  | [2022-09-09 08:45:23.777] [info] [progress] Linear Regression: predict: post-processing (2/2)
computation_container1  | [2022-09-09 08:45:25.220] [info] [Predict]                      End grpd

logistic regression

  • use internal 1k sample data
  • log 
    computation_container1  | [2022-09-09 09:12:21.061] [info] method id is: 6
computation_container1  | [2022-09-09 09:12:21.394] [info] [progress] SGD optimize: remain: 100
computation_container1  | [2022-09-09 09:12:23.478] [info] [progress] SGD optimize: remain: 99
computation_container1  | [2022-09-09 09:12:24.741] [info] [progress] SGD optimize: remain: 98
...

computation_container1  | [2022-09-09 09:14:23.755] [info] [progress] SGD optimize: remain: 0
computation_container1  | [2022-09-09 09:14:23.993] [info] end_job_id is 2
computation_container1  | [2022-09-09 09:14:24.351] [info] [Predict]                      Received
computation_container1  | [2022-09-09 09:14:24.351] [info] Model Manager: Model Id is 2
computation_container1  | [2022-09-09 09:14:24.594] [info] [progress] Logistic Regression: predict: pre-processing (0/2)
computation_container1  | [2022-09-09 09:14:24.793] [info] [progress] Logistic Regression: predict: compute (1/2)
computation_container1  | [2022-09-09 09:14:27.990] [info] [progress] Logistic Regression: predict: post-processing (2/2)
computation_container1  | [2022-09-09 09:14:28.398] [info] [Predict]                      End grpd

come2ry
come2ry requested changes Sep 9, 2022
View reviewed changes
@arukuka arukuka changed the title Show progress Add simple progress log Sep 9, 2022
@arukuka arukuka marked this pull request as ready for review September 9, 2022 11:08
@come2ry come2ry merged commit f6ff1b5 into main Sep 11, 2022
@come2ry come2ry deleted the feature/otsuka/add_progress_log branch September 11, 2022 16:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@come2ry come2ry come2ry approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@arukuka @come2ry